JVM memory allocation and common methods String

A, JVM memory allocation and constant pool

Before introducing the String class, first a simple analysis of the JVM, use of memory is how the allocation. As shown below (note: there is no method jdk1.8 after the region):

As JVM memory is divided into different zones, which have their uses, and create time destroyed, some areas with the virtual machine process exists, some regions are dependent on the thread start and end users to to create and destroy.

Description area names:

1.1, the method area:

Belonging to the shared memory area for storing virtual machine has been loaded class information, constants, static variables, the time compiler to compile the code and other data.

1.2, the virtual machine stack

VM stack is what we usually say that the stack, Java memory model is the method of execution, each execution of a method, you create a stack frame. When the stack frame onto the stack, the return to normal when the result or exception capture Java method invocation stack the stack frame.

Frame stack: the stack frame information storing method, comprising the local variable table number, return values, operand stack, dynamic linking.

1.3, native method stacks

In terms of functionality and virtual machine stack is similar, but the virtual machine stack byte code is executed, and the local stack method is called Native method, and it is thread-exclusive.

1.4, the program counter

The program counter is exclusive thread, which is the line number record bytecodes thread of execution. When multi-threaded execution, CPU will switch back and forth between the threads, then when one thread back again, that is how the thread storage unit and executing instructions. And an address of the program counter will be stored in the storage unit, after the implementation of the program counter is automatically incremented by 1, until it terminates this cycle.

1.5, heap

When it comes to the concept of heap presumably are not familiar, it is an important role in memory. It is mainly used to store object is created, instances out by keyword new, it is one of the largest areas shared by all threads.

== Important Note: JDK1.7 and later, the constant pool to a heap memory. ==

Further comprising a stack == == constant pool for storing generated during compilation == == literal references and symbols. This section after the class is loaded, it is stored in the method area. Meanwhile, new constants generated runtime constant pool may be placed, such as in a class String intern () method produces a constant.

Constant pool is a constant of the type used in the ordered collection. Including direct constant (basic types, String), and references to other types of symbols, methods, fields.

Second, the constant pool

2.1 What is constant:

Constant refers to final modified variable value can not be changed once it is determined.

static final variables can be modified, methods, instance variables and local variables.

The constant pool is divided into two types: static constant pool and runtime constant pool

2.2, static constant pool

* .Class file that is in constant pool, class file constant pool not only contains the string (digital) literal, also contains the class, method of information, most of the space occupied by the file class. This constant pool for storing the literal and symbolic reference amount.

2.3, runtime constant pool

Refers to a virtual machine JVM class loading operations after the completion of the class file constant pool is loaded into memory and stored in the method area, we often say that the constant pool, method of operation is the time constant pool area. A significant amount of pool often run the same feature is dynamic, means that not only will produce constant need at compile time, will save the new constant in the constant pool to run, such as the String class intern () method.

Third, the == and equals

3.1 area between both sides of


For basic types is: a comparative value ==

For reference types is: == shows a comparison of address values


The comparison is between the two values ​​are equal, but in Java classes are derived directly or indirectly Object class, but also equals no exception. In fact, the source code is the use of equals == comparison, the following source code:


So the question is, and this == What difference does it?

Mentioned above equals it is inherited from java.lang.Object, and therefore can be rewritten to define the equals our own way of comparison.

See the following code:

        String str1 = "abc";
        String str2 = "abc";

        char[] strArray = {'a','b','c'};
        String str3 = new String(strArray);

        String str4 = "abc";

        System.out.println(str1 == str2);   
        System.out.println(str1 == str3);   
        System.out.println(str2 == str3);   

The above operating results as follows:


Next, we turn the above analysis results:

1, str1 str2 comparison with the target address is a string, because their values ​​are the same, the address value is also the same.

2, str3 example is the new out of the objects in the heap memory will open up a new memory address, it is not in the constant pool. It returns a value of false.

3, empathy and str2 str3 comparison is the same result.

4, equals the comparison values ​​are identical, the result returned is true.

as the picture shows:

Four, String commonly used method

First statement string:

 String str1 = "abc";

4.1,int length()

 int length = str1.length();

4.2, char charAt (value)

String str= "abc";
char c = str.charAt(1);  

4.3,char toCharArray()

String str= "abc";
char c[] = str.toCharArray();  
for (int i = 0; i < c.length; i++) {
      System.out.println("转为数组输出:" + c[i]);

4.4, int indexOf ( "character"); int lastIndexOf ( "character")

String str="axcdefgabc";
int a1 = str.indexOf("a"); 
int a2 = str.indexOf("x",  2);
int a3 = str.lastIndexOf("c"); 
System.out.println("你的位置为:" + a1); 
System.out.println("为的位置为:" + a2);
System.out.println("点最后出现的位置为:" + a3);

4.5, string case conversion

toUpperCase (); uppercase
    the toLowerCase (); converted to lower case

String str = "hello world";
String str1 = "HELLO WORD";
System.out.println("将字符串转大写为:" + str.toUpperCase());
System.out.println("将字符串转换成小写为:" + str1.toLowerCase());

4.6, String [] split ( "character")

String str = "abc,def,123";
String[] arr1 = str.split(",");

4.7,boolean equals(Object anObject)

 String str = "abc";
 String str1= "123";  
 if(str.equals(str1)) {  

4.8,String trim()

String str = "       abc         ";  
System.out.println("去掉左右空格后:" + str.trim());

4.9 string replacement

String replace (char oldChar, char newChar)
    String replaceAll (String, String) will replace all the contents of a specified content
    String repalceFirst (String, String) will be one of the first occurrence of the contents replaced with the content specified

String str = "abcdefgabdc";  
System.out.println("替换:" + str.replace("abc", "123")); 
System.out.println("替换全部:" + str.replaceAll("ab", "12")); 
System.out.println("替换第一次出现:" + str.repalceFirst("a", "a")); 

4.10,String substring(int beginIndex,int endIndex)

 String str = "abcdefg";  
 // 截取0-3个位置的内容, 不含3 
 System.out.println("截取后的字符为:" + str.substring(0, 3));
 // 从第3个位置开始截取, 含2
 System.out.println("截取后字符为:" + str.substring(2));

4.11,boolean equalsIgnoreCase(String)

String str = "ABC";
String str1 = "abc";

4.12,boolean contains(String)

String str = "ABCDEF";
String str1 = "ABC";

V. Summary

1, to the dispensing JVM memory, there is a method in jdk6 region, there is no method in jdk8 region, into cell regions.

2, there is a method in jdk6 constant pool area, after the move to the stack jdk7 constant pool.

This blog is simply introduces the concept of JVM memory allocation and constant pool, there is still a lot more in-depth principle not recorded. Therefore, the latter will continue to update the knowledge of the principles of the relevant JVM, the above does not apply if the blog also please leave a message (email) advice.

Thanks for reading!


Elasticsearch (4) — basic concepts (Index, Type, Document, clusters, nodes, and a copy of the fragment, inverted index)

Elasticsearch core technology (2) — Basic Concepts

This blog talked about the basic concepts include: Index, Type, Document. Clusters, nodes, and a copy of the fragment, inverted index.

A, Index, Type, Document


index: Index is a document (Document) containers, is a kind of collection of documents.

The word in the index ElasticSearch have three meanings:

1), index (noun)

Analogy traditional relational database field, the index is equivalent to a database (Database) SQL is. Index by its name (must be all lowercase characters) are identified.

2), the index (verb)

Save a document to process index (noun) a. This is very similar to the SQL statement INSERT keyword. It is equivalent to UPDATE the database if the document already exists.

3), inverted index

Relational database by adding a B + tree index to the specified column to enhance the speed of data retrieval. ElasticSearch index uses a structure called the inverted index to achieve the same purpose.


Type can be understood as a relational database Table.

Previously, the concept of a document index and middle there is a type under each index can set up multiple types, you need to specify the index and store the document type. 6.0.0 From the beginning there is only a single index type,

7.0.0 The future will not recommended, after 8.0.0 do not support.

The reason abandoned the concept:

Although we can go to popular understanding Index for SQL Database ratio of, Type Table likened the SQL. But this is not accurate, because if in SQL, independent of each other before the Table, the field of the same name has nothing in both tables.

But in the ES, under the same Index Different Type If there is a field of the same name, they will be treated as Luecence the same field, and they must have the same definition. So I think now more like a table Index,

The Type field did not have much significance. Currently Type has been Deprecated, beginning at 7.0, an index can only be built as a Type _doc


Document Index recorded a single which is called Document (document). Equivalent relational database table rows.

We look at a document source data

_index document belongs index name.

_type document belongs type name.

_id primary key of Doc. When written, the ID value can be specified Doc, if not specified, the system automatically generates a unique UUID value.

Version Information _version document. Elasticsearch by using the version to ensure that changes to the document can be performed in the correct order to avoid data loss due to out of order.

_seq_no strictly increasing sequence numbers, one for each document, Shard level strictly increasing, Doc’s _seq_no written after the guarantee is greater than Doc’s _seq_no first written.

primary_term _seq_no and also as an integer primary_term, whenever Primary Shard occur reassigned, such as restart, Primary elections, _primary_term incremented by 1

ID found query correctly so ture, if Id is not correct, finding out the data, found field is false.

JSON data _source original document.

Second, clusters, nodes, and a copy of the fragment

1, cluster

ElasticSearch cluster is actually a distributed system, it needs to have two characteristics:

1) High Availability

a) Service Availability: Allow nodes out of service;

b) availability of data: part of the node is lost, the data is not lost;

2) Scalability

With the rising amount of requests, the growing amount of data, the system data can be distributed to other nodes, to achieve horizontal scaling;

A cluster can have one or more nodes;

Cluster health value

    green: All major fragmentation and fragmentation are available copy

    yellow: All major slice is available, but not all copy fragments are available

    red: Not all of the major fragments are available

当集群状态为 red,它仍然正常提供服务,它会在现有存活分片中执行请求,我们需要尽快修复故障分片,防止查询数据的丢失;

2, the node (Node)

What 1) nodes are?

a) node is an example of a ElasticSearch, which is essentially a Java process;

b) ElasticSearch can run multiple instances on a machine, but it is recommended to run a ElasticSearch instance only on one machine in a production environment;

Node is a single cluster of servers, for storing data and providing search and indexing cluster. Like cluster node also has a unique name, the default when the node startup will generate a uuid as a node name,

The name can also be specified manually. Single cluster may be composed of any number of nodes. If only one start node, a single-node cluster will form.

3, slice

Primary Shard (primary slice)

ES shard to solve the problem of node size limit ,, the main fragments may be distributed over the data to all nodes in the cluster.

The relationship between them

 一个分片是一个lucene index(此处的index是lucene自己的概念,与ES的index不是一回事);

Main distributor is specified when the number of sheets index creation, can not be modified follow, unless Reindex

Index data stored in a plurality of sub-sheets (as a default), equivalent to the level sub-table. A slice is a Lucene instance, it is itself a complete search engine. Our documents are stored and indexed into fragments.

But the application is interacting directly with the index rather than fragmentation.

Replica Shard (copy)

There are two copies of important roles:

1. Service availability: Because the data only one, if a node linked, and that the presence of the above data is all lost, with replicas, as long as this data is not stored in the node trailer, the data will not be lost. So do not copy and fragmentation

Primary slice allocated to the same node;

2, scalability: to improve search performance by a parallel search across all replicas because the data on the replicas are near real-time (near realtime), so all replicas can provide search capabilities, by setting reasonable replicas.

You can increase the number of high throughput search

Setting slice

Carved piece set for a production environment, we need to do capacity planning in advance, because the main points is the number of pieces in the pre-set index creation, follow-up can not be modified.

It is set too small number of fragments

Lead to subsequent nodes can not increase the level of expansion.

Resulting in the amount of data pieces is too large, time-consuming data reallocation;

Setting number is too large fragment

Affect the search results of relevancy scoring, affect the accuracy of statistical results;

On a single node excessive fragmentation, it would lead to waste of resources, but also affect performance;

Third, the inverted index

ES search function is based on the fundamental principle of lucene, but lucene search index is a flashback, with the reverse order of the results related to the type of word.

For example

1, assume a collection of documents comprising five documents, documents every content As shown in FIG leftmost column is a text block corresponding to each document ID.

As shown (FIG. Pirates)

2, the first to use word system will automatically cut documents into word sequence record which documents containing the word, at the end of such a process, we can get the most simple inverted index.

3, the indexing system may also record additional information in addition, also described under FIG word frequency information. The document is divided into one sentence term (term used to indicate a word or words, depending on the method used word),

Flashback index stored in the term, term frequency of occurrence (tf, term frequency) and the emergence of location (flashback index words are in order, and this figure does not reflect it), please note that the contents of this document are document

In a field that is indexed each field has its own flashback index

A simple search process

Suppose we search Google Maps father, the search process will be the case

    Word, sentence segmentation plug-in will be divided into three valleys term song, map, father

    These three get flashbacks term index to find (would be very efficient, such as binary search), if matched to the corresponding document id Take obtain the document content

However, how to determine the order of results?

Here to introduce the concept of _score, for the term of the match, lucene will have on its score, the higher the score, more high ranking here to introduce several related concepts

- TF(term frequency),词频,term在当前document中出现的频率,一个term在当前document中出现5次要比出现1次更相关,打分也会更高
- IDF(inverse doucment frequency),逆向文档频率,term在所有document中出现的频率,这个频率越高,该term对应的分值越低
- 字段长度归一值,简单来说就是字段越短,字段的权重越高, 比如 term `我`在匹配 `我123`和`我123456`时,`我123`的得分会更高.


1, Elasticsearch core technology and combat — Ruan Yiming (eBay Pronto platform technical director

2, ElasticSearch basic concepts

3, Elasticsearch the basic concepts

4, ElasticSearch Section 5 inverted index, tokenizer



spring loaded bean flow analysis

spring framework as a basis for the development of our current daily basic development work and his inseparable, as most classic bean management, excellent framework, its complexity is often prohibitive. But as a framework for daily life, we have to understand a problem that spring is how to load the bean, notes that we often use in development such as @ Component, @ AutoWired, @ Socpe such as annotation, how Spring is resolved, understand these principles will help us to a deeper understanding of the spring. It should be noted that the source is spring very sophisticated, complex, limited to limited space, this blog will not be detailed analysis of the source code, will take a careful look at the way, avoid the light weight, seize the key to the entire process analysis (analysis does not specific details), this will be based on the version spring5.0

This blog directories:

A: spring configuration or reading process annotations

Two: bean life cycle of spring

Three: spring of BeanPostProcessor processor

Four: Some key issues

Five: Testing

Six: summary

A: spring configuration or reading process annotations

1: first specify the annotation by scanning the spring package path, such as spring identification @ Component, @ Service, @ Lazy @Sope attribute xml or other annotations configured (by reading the stream into a parse Document, Document) will then spring resolve these properties, these properties BeanDefintaion encapsulates the implementation of the interface class.

In springboot, we can also be used to configure the way notes:

This configuration example Bean, spring will also className, scope, lazy, etc. These properties are fitted into the corresponding BeanDefintaion PersonAction in particular uses BeanDefinitionParser interface parse (Element element, ParserContext parserContext) method, this interface there are many different implementations class. Notes to parse through the implementation class or xml and then put BeanDefination, the role BeanDefintaion that integrates a variety of attributes our configuration objects, it is important to have this bean is ClassName, if there is Singleton, attributes and values ​​such as object (if single embodiment, it will be behind the singleton object into a single cell embodiment of the spring). If you need late spring directly from these properties will be in it. Then, re-registration to a ConcurrentHashMap, the spring in particular is to registerBeanDefinition (), the Map stored key is the name of the object, such as Person object, its name is the person, the value is BeanDefination, which is located DefaultListableBeanFactory class below the beanDefinitionMap class attribute while all put into beanDefinitionNames bean name in the list, the purpose is easy to take the beanName;

Two: bean life cycle of spring

In fact, the life cycle of bean spring core is divided into four steps, as long as the three key steps to sort out, others just add different details to achieve in these three details, that is spring of bean Shengming cycle:

The difference between instantiation and initialization: instantiation in jvm heap created the object instance, this time it’s just an empty object, all the property is null. And after the initialization process is to talk about some of the properties of the object-dependent assignment, call some method to open some default load. A database configured such spring properties Bean, at initialization time will fill these properties, such as driver, jdbcurl the like, and then initiates the connection

2.1: instantiation Instantiation

AbstractAutowireCapableBeanFactory.doCreateBean will call createBeanInstance () method, which is the main phase from beanDefinitionMap bean read cycle, acquiring its properties, and the use of reflection (ReflectionUtil there will first force the constructor setAccessible (true the core package)) object to be read constructor (spring will automatically determine if there is no parameter or parameters, and the constructor parameter is available), and then to create an instance (the newInstance)

2.2: Initialization

Initialization consists of two steps, a property is filled, the other is a specific initialization process

2.2.1: property assignment PopulateBean () will depend on the properties of the bean filling, @ AutoWired annotation injected properties occurs at this stage, if we have a lot of bean depend on, then spring will in turn call these dependent objects instantiated of note here may have a circular dependency problem. Later we will talk about how to solve the spring is a circular dependency problem

2.2: Initialization Initialization

The initialization process includes the initialization good bean to spring into the cache, filling our pre attributes do further post-processing, etc. 3: use and destruction of Destruction

After all Spring bean initialization are good, our business systems can be called up. The destruction of the main operation is the destruction of bean, the main spring is accompanied by closed container, this time will be removed in spring of bean container. After this spring’s life cycle to the end of this step is complete, no longer accept the management and restraint spring.

Three: spring of BeanPostProcessor processor

Another power of spring is to allow developers to extend the definition of self-initialization process the bean, the main idea is realized by BeanPostProcessor achieved, spring a variety of pre- and post-processors that permeate created in bean the Ins and Outs, interspersed at various stages of the life cycle of spring, every step will affect the spring of bean loading process. Next we have to analyze specific process:


3.1: Examples stage

Examples of, after performing the instantiation, the calls InstantiationAwareBeanPostProcessor postProcessBeforeInstantiation method calls the empty stage of object’s constructor object

The BeanPostProcessor (the specific implementation is InstantiationAwareBeanPostProcessor) postProcessBeforeInstantiation ().;

This stage allows the previous instantiated in Bena, allowing developers to custom logic, such as returning a proxy object. But note that if returned at this stage is not a null instance of, spring break will be the follow-up process. BeanPostProcessor.postProcessAfterInstantiation ();

This stage is performed after the post-processing operation is completed Bean instantiated, executed before all initialization logic, the logic assembly

3.2: initialization phase


This method is called before the bean initialization method Spring AOP infrastructure is also performed by the proxy logic is implemented BeanPostProcessor


Custom property value of the process allows us to set object attributes, if some businesses, some attributes of an object is null, but can not be displayed, or other fixed value 0 is null, such as display, we It may be implemented in the null value into a particular value in this method

3.2.3: BeanPostProcessor.postProcessAfterInitialization (Object bean, String beanName). The process may be performed after the bean instantiated in this method, such as our custom annotations, versioning of objects dependent automatic route switching. For example, there is a service depends on two versions of the implementation, how do we achieve it automatically switches? This time can customize a routing annotation, if called @RouteAnnotaion, and then implement BeanPostProcessor interface, then the set of routing rules therein annotation @RouteAnnotaion get customized by reflection.


4.1: launch the operational phase of the vessel


Container formal rendering is complete, begin the startup phase, bean container has been under the management of spring, the program can call at any time

5.1: Stop the destruction of container 5.1.1: SmartLifecycle.stop (Runnable callback)

spring container stops running 5.1.2: DisposableBean.destroy ()

spring will destroy all of the bean, when the bean instance is destroyed to achieve the release of resources is called

Four: Some key issues

4.1: FactoryBean and the difference between the BeanFactory?

BeanFactory a bean factory class interface, is responsible for the production and management of bean plants, is the bottom IOC container and base interface, spring used to manage and assembly common bean’s IOC container, it has a variety of implementations, such as AnnotationConfigApplicationContext, XmlWebApplicationContext etc. .

FactoryBean FactoryBean is a spring bean belongs, based on the IOC container to achieve a simple adding Bean factory mode and decorative patterns, can be produced is a factory object and decorative objects bean, managed by the spring, it is produced by the object determined by the getObject () method. Note: It is generic, only the production of a type of object is fixed, unlike BeanFactory can be produced as a plurality of types of Bean. In Bean for some special handling, such as Bean itself is a factory, then a separate instance of the operating logic in it, and we may not want to leave the set of logic spring, then you can implement the interface yourself FactoryBean control logic.

4.2: spring how to resolve circular dependencies

Circular dependencies is A-> B-> A, spring in the creation of A, B relies found, because to create B instance, depend on the A and B found, went to create A, because a closed loop is formed, can not be stopped it could lead to soaring computing cpu

How to solve this problem? spring solve this problem mainly rely on clever three cache, cache mainly refers to the so-called three map, singletonObjects main storage is a singleton object that belongs to the first-level cache; singletonFactories belong to singleton factory object, it belongs to the third level cache ; earlySingletonObjects belongs to the second-level cache, how to understand the early identification of it? It represents just after object instantiation has not been initialized. Spring first try to obtain from singletonObjects (cache), if the object is created and can not get in, try to obtain from earlySingletonObjects (secondary cache), and if you still can not get permit from singletonFactories acquired by getObject, through singletonFactory.getObject () (three-level cache) acquisition. If you get to the corresponding singletonFactory is then removed and placed into the singletonObject earlySingletonObjects, in fact, will be upgraded to level three cache secondary cache, the cache is upgraded. During the spring when the object is created, it will in turn look for objects from primary, secondary and tertiary cache, if found direct return. Because it is first created, can only be found from the third level cache (instantiate go into the stage), create an instance of complete, then the cache into the first level cache. Then the next cycle dependent from a cache can get directly to the instance object.



Five: Testing

Let’s write a class test, verify the above questions:

5.1: First declare a custom Bean

public class CustomBean {
    public CustomBean(){

5.2: Declaring a Bean to achieve BeanPostProcessor

package com.wyq.spring.bean;

import org.springframework.beans.BeansException;
import org.springframework.beans.PropertyValues;
import org.springframework.beans.factory.DisposableBean;
import org.springframework.beans.factory.SmartInitializingSingleton;
import org.springframework.beans.factory.config.BeanPostProcessor;
import org.springframework.beans.factory.config.InstantiationAwareBeanPostProcessor;
import org.springframework.context.annotation.Scope;
import org.springframework.stereotype.Component;

import java.beans.PropertyDescriptor;

 * @Author: wyq
 * @Desc:
 * @Date: 2019/9/1 15:36
public class TestBean implements BeanPostProcessor, SmartInitializingSingleton, InstantiationAwareBeanPostProcessor, DisposableBean{

    private static final String BEAN_NAME= "customBean";

    public Object postProcessBeforeInitialization(Object bean, String beanName) throws BeansException {
        if (BEAN_NAME.equals(beanName)) {
        return null;

    public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException {
        if (BEAN_NAME.equals(beanName)) {
        return null;

    public void afterSingletonsInstantiated() {


    public Object postProcessBeforeInstantiation(Class beanClass, String beanName) throws BeansException {
        if (BEAN_NAME.equals(beanName)) {
        return null;

    public boolean postProcessAfterInstantiation(Object bean, String beanName) throws BeansException {
        if (BEAN_NAME.equals(beanName)) {
        return false;

    public PropertyValues postProcessPropertyValues(PropertyValues pvs, PropertyDescriptor[] pds, Object bean, String beanName) throws BeansException {
        return null;

    public void destroy() throws Exception {

5.3: Start container:

Six: summary

This blog is to introduce some examples of the process of Spring, analysis of what strategically advantageous position of bean loading spring, and did not begin a detailed analysis of the details. spring internal source is very complex, each interface implementation class in more than five, if the in-depth details, a blog is probably not able to speak clearly. The purpose of this blog is in the basic context of the center line of the sequence set forth in spring, first of all we need to have a general understanding, then down to the details is easy. This is also a learning methodology, through this blog I hope to sort out a clear spring of the basic processes of spring will have a clearer understanding. Excellent open source framework and learn to design basic idea there is to further enhance their ability to read the source code.



Use of the six filter frame before and after the end of the separation structures scratch NetCore2.2 (EF Core CodeFirst + Autofac) + Vue item verification request data globally

He talked about how the global exception handler in NetCore project in the last article, when an interceptor processing system or manually throws an unhandled exception occurs conducted.

In this section mentioned API request a verification of the model, to throw a few questions,

    Why use model validation? Will produce some unexpected errors to my understanding, the effective average user data is not entered, it is possible to use these data in an application.

    what’s the effect? Use model validation is to ensure that requested data can be effectively used in the program, but also to avoid some unusual cases, or that you can not go to the correctness of the data in a relational model interface code, as has been verified by the model.

    how to use? MVC model validation provides better support, provides many features, you can set validation rules by Model metadata, with ModelState to handle error messages, get error messages.

Provides many built-in features in ASP.NET Core MVC in, what are some of the more commonly used built-in features:

Here are some built-in validation features:

    [CreditCard]: Verify whether the property has a credit card format.

    [Compare]: verification models two attributes match.

    [EmailAddress]: Verify whether the property has e-mail format.

    [Phone]: Verify that the property has the phone number format. Looks like we can not use this number, it is recommended to use regular properties

    [Range]: validating property values ​​are within a specified range.

    [RegularExpression]: validate the attribute value is specified regular expression matching.

    [Required]: Verify field is non-NULL.

    [StringLength]: verification string attribute value does not exceed the limit specified length.

    [Url]: Verify whether the property has a URL format.

    [Remote]: by the process operation on the call server, the authentication client input.

In addition to these built-in features can also add custom properties. For convenience example, create a custom attribute ClassicTestEqualAttribute verify field value is equal to the value built

    /// 验证值是否等于内置值
    public class ClassicTestEqualAttribute : ValidationAttribute
        private string _bulitIn;
        private string _cusValid;

        public ClassicTestEqualAttribute(string bulitIn)
            _bulitIn = bulitIn;

        protected override ValidationResult IsValid(
            object value, ValidationContext validationContext)
            var movie = (TestValidModel)validationContext.ObjectInstance;
            _cusValid = movie.CusValid;
            var cusValid = (string)value;//

Two ways to obtain the field value - value of another field may be acquired

if (_bulitIn != cusValid) { return new ValidationResult(GetErrorMessage()); } return ValidationResult.Success; } public string CusValid => _cusValid; public string GetErrorMessage() { return $"

Test model built {_bulitIn} is not equal to the value

"; } }

Then create a test model for testing

    public class TestValidModel
        [Required(ErrorMessage = "

Please enter name

"), RegularExpression("^(?!_)(?!.*?_$)[a-zA-Z0-9_]{4,12}$", ErrorMessage = "

Login does not conform to the rules, enter a 4-12 bit data does not contain special characters

")] public string Name { get; set; } [Range(22,35,ErrorMessage = "

Age should be between 22-35

")] public int Age { get; set; } [ClassicTestEqual("test")] public string CusValid { get; set; } [StringLength(3,ErrorMessage = "

In length for the character string within 3

")] public string Role { get; set; } [Required(ErrorMessage = "

Please enter the message

")] [EmailAddress(ErrorMessage = "

Incorrect format

")] public string Email { get; set; } [Required(ErrorMessage = "

Please enter the phone

"), RegularExpression("^1[3|4|5|6|7|8|9][0-9]{9}$", ErrorMessage = "

Phone number is not formatted correctly

")] public long Phone { get; set; } }

This time to add a test method, and then call interface by Postman, call our body when nothing Chuan try. PS: Postman is an excellent interface test software.

Here we can see the model validation has failed, and the number of errors as to why only five, because has been restricted in the Startup category, the maximum number of validation errors only five.

services.AddMvc MaxModelValidationErrors to change.

services.AddMvc(options =>
                options.MaxModelValidationErrors = 5;//

The maximum number of validation errors

options.AllowValidatingTopLevelNodes = true;//

Verify whether to allow the top node interface method parameters

options.Filters.Add(new ExceptionFilter());//

Add exception processing filter


When an incoming request data interface to meet the requirements of model verification will pass.

Here it is available from ModelState in an error message, but to demonstrate the effect of model validation would not write. You can not always be done when model validation are so to write code like this again, so we use filters to request global model validation.


First, we add model validation methods, inherited property ActionFilterAttribute

    /// 模型数据验证
    public class ModelValid : ActionFilterAttribute
        public override void OnActionExecuting(ActionExecutingContext filterContext)

Determining whether the model is verified by

if (filterContext.ModelState.ErrorCount == 0 && filterContext.ModelState.IsValid) return; var errMsg = new StringBuilder(); foreach (var modelStateKey in filterContext.ModelState.Keys) { var value = filterContext.ModelState[modelStateKey]; foreach (var error in value.Errors) { if (!string.IsNullOrEmpty(error.ErrorMessage)) { errMsg.Append(error.ErrorMessage + ","); } } } if (errMsg.Length > 0) { errMsg.Remove(errMsg.Length - 1, 1); } if (filterContext.Controller is BaseController controller) filterContext.Result = controller.Fail(1005, $"

Request data fails verification, {errMsg}


Error Code 1005 is a custom

else throw new CustomSystemException("

Default Controller should inherit BaseController, in order to achieve a global model validation, error remind


Error Code 999 is a custom

} }

Then the model validation filter property is added to the base class BaseController controller, because each controller will inherit the controller, and this model validation filter triggers on every request.


After this addition was complete model will be validated when accessing each interface. At this time of the previous model test method to determine commented out, then re-run the test access interface facie effect?

There’s only complete when the model validation process to go through the business logic will. Otherwise it will not be implemented business model will directly return the validation error messages. PS: The above is worth saying is, why did not lose a phone number, will not return “Please enter a phone number.” Because long type is not NULL, so when there is no input phone numbers, phone field is the default, namely Phone = default (long) // 0 Only long type into long? Or can at no time Nullable trigger Required characteristics enter a phone number. As it has been changed to be the empty type.

Above is carried out before performing verification model, we can also verify that the data model to meet the requirements again after performing a series of business logic, as long as the execution TryValidateModel (validModel); can be.

Test interface method into the following:

        public ActionResult ValidModel([FromBody]TestValidModel validModel)
            //            if (!ModelState.IsValid)
            //            {
            //                return Fail(1005, "模型验证失败");
            //            }
            validModel.Age = 18;
            var b = ModelState.IsValid;
            return Succeed("


"); }

Then again visited by the above model of success, debugging break point can be verified result is false, in fact, in this place can be re-validated method to the base class to the controller, if the validation fails, the validation fails direct return results.

In addition to model validation, verification can be top-level node, which is the interface method parameter on direct verification. But generally not recommended for this use. You have to do a little change in the Startup category, above the code has been posted out. Sample code is as follows:

        [AcceptVerbs("Get", "Post")]//

Allow get and post method

[Route("validphone")] public ActionResult TestValid([Required(ErrorMessage = "

Please enter your phone number

")] [RegularExpression(@"^1[3|4|5|6|7|8|9][0-9]{9}$", ErrorMessage = "

Not a valid phone number

")] string phone) { //phone = "123"; //TryValidateModel(phone);//

The single field-based methods should be used invalid

if (!ModelState.IsValid) { var errMsg = new StringBuilder(); foreach (var modelStateKey in ModelState.Keys) { var value = ModelState[modelStateKey]; foreach (var error in value.Errors) { errMsg.Append(error.ErrorMessage + ","); } } errMsg.Remove(errMsg.Length - 1, 1); return Fail(1, errMsg.ToString()); } return Succeed("


"); }

Implementation of the results is not a map, as long as everything after his own attempt will understand.


The next one will be how to generate NetCore by jwt token, and token authentication, refresh and other personal thoughts.


Need source code under comments or private letter to me ~ SVN guest account password to download the code is not on GitHub.



Intelligent Control and gateway system implementation (project)

Date: June 2019

Stage: junior next semester

Background: Things Gateway development technology & network engineering design class-based Design and Implementation

Name: Intelligent Control and gateway system implementation


GEC 6818 Guangdong embedded board as the board gateway system, which is a central data system. Start serial port transceiver, ZigBee coordinator may receive network data transmitted through the serial port; lCD screen programming, real-time display of data sensor network; MySQL database to establish a connection, and the temperature and humidity sensor networks door state information may be saved to a remote database server; to establish a TCP server, the communication can be achieved with the application of the upper layer; connected through the serial port NB-IOT module, data forwarding for external traffic.

Things of perception layer using a ZigBee network, a plurality of terminal nodes to achieve the sensor data acquisition, the control device of the plurality of terminal nodes, the unified management data by the coordinator; the network layer 6818 is embedded Guangdong development board, integrate multiple protocols, to achieve data forwarding process; WEB application end has an end-side data display and lamp control handset.

Keywords: NB-IOT; IOT gateway; database; TCP / IP; MQTT


1.1 Description topic

Compared to the age of the Internet, networking communication protocols more diverse, fragmented matter is very serious, thus highlighting the importance of the gateway will – Things Gateway able to integrate different information was collected, and it transmission to the next level, and thus are unable to transmit information between the various parts. Things aware network gateway may be implemented with a communication network, and a protocol conversion between different types of network-aware; both wide area network, a local area network can also be implemented.

Things gateway, as a new term in the coming era of things will play a very important role, it will be perceived link between the network and traditional communication network. Further Gateway also need to have things device management capabilities, operators can manage things by each gateway node underlying perception, understanding of the relevant information of each node, and remote control.

1.2 target system

Things of perception layer using a ZigBee network, a plurality of terminal nodes to achieve the sensor data acquisition, the control device of the plurality of terminal nodes, the unified management data by the coordinator; the network layer 6818 is embedded Guangdong development board, integrate multiple protocols, to achieve data forwarding process; WEB application end has an end-side data display and lamp control handset.

1.3 Environment

a) Operating System:

Windows10 Education Edition 64


b) software:

Android Development: Android Studio2.0

Web developers: Adobe Dreamweaver CC 2017

Database: MySql

Document write: Microsoft Word 2016

Serial command line login: putty.exe

Tftp server: tftpd32

c) the development of language:

Java language

PHP language

C language

System design and detailed design

2.1 System Design

FIG. 2.1 System Design FIG.


2.2 Gateway board design:

Things Gateway development board using the 6818 Guangdong embedded development board, GEC6818 development platform, core board 10 laminate process design, to ensure stable and reliable, can be used for batch tablet PCs, car machines, machine learning, POS machines, game consoles , monitoring industry and other fields. The platform is equipped with Samsung’s Cortex-A53 series of high-performance eight-core processor S5P6818, the highest frequency up to 1.4GHz, it can be used in embedded Linux and Android-driven, application and development of the operating system. Development board left a wealth of peripherals, support for Gigabit Ethernet, onboard LVDS interface, MIPI interface, USB interface.

GEC6818 development platform supports the three major operating systems, with full teaching resources and teaching content, including: ARM microprocessor system-driven experiments, embedded real-time operating system Linux development, embedded systems development Android, Android embedded application development, embedded content combat systems project development.

GEC6818 development board using the embedded Linux system, running inside the TCP server, the client may wait for TCP connection to receive control commands and data submitted; internal database clients running, the sensor data can be stored aware layer reported to the database server; 800 * 480 integrated with the board screen, the display module can be achieved by the sensor data and the status display screen; board by two serial ports, respectively aware layer ZigBee network coordinator, NB-IOT module traffic. Integrated, multi-protocol data transmission and reception, the process forwards.

GEC6818 development board is the central data system.

FIG board design Gateway 2.2

2.3 ZigBee network aware layer:

Level sensing system using a ZigBee network data transmission network, which has a coordinator and two end sections. And sensing node implements a control terminal, which includes sensing portions are temperature and humidity sensors, Hall gate sensor; controlling part is distributed on the three-color LED lights in each terminal.

FIG. 2.3 ZigBee network aware layer

2.4 display control platform and data storage:

The display control panel platform is based on a gateway for users to develop, it would require the data gateway after the process returns to the user by a simple way of viewing and other graphics operations, the use of fast data. Gateway 6818 is connected with the LCD display panel, the display state of the door switch to achieve, two upper and lower end of the display modules, real-time display of temperature and humidity. TCP display console, is achieved by a gateway connecting plate TCP server, by acquiring the transmitted data gateway plate, the display state analyzing module, door status, temperature and humidity data can also be achieved under control commands is triggered by a button hair. Data will be stored gateway receives the sensor data is saved to the database in real time, the data show a database query will show through the website. Data receiving and transmitting the control command through the mobile phone terminal MQTT protocol, the protocol gateway plate MQTT achieved by NB-IOT module.

Figure 2.4 shows the control and data storage platform

2.5 NB-IOT module implements:

Receiving sensor data associated with the light transmission control instruction a communication gateway, an application protocol NB communication module, to realize the application layer and the application layer MQTT phone through the serial port module NB. MQTT server is set up on Tencent cloud host.

BC26 features:

Full Netcom

AT command support

Support OpenCPU, eliminating the need for external MCU

Mobile support high-speed (80km / h ~ 120km / h)

Use MediaTek’s MT2625 chip


1. OpenCPU scheme is based on the secondary development Quectel module, the user can develop applied directly integrated in the module, thus eliminating the external master MCU. OpenCPU has been widely used in M2M fields, such as the field of smart home, smart cities, asset tracking, automotive and energy.

2. OpenCPU comparison with the conventional scheme shown below. In the conventional scheme, it requires an MCU as a main controller for controlling hardware peripherals, such as control of lights on or off, then the MCU standard UART interface with the communication module AT command, report the current status lights, or receive from a remote server the lamp command switch. And OpenCPU embodiment, the module can be written directly in the program control lights, save hardware costs and speed up application development.

3. OpenCPU core system of hardware and software modules NB-IoT binding. It is built ARM Cortex-M4 processor FreeRTOS built on top of an operating system, having a micro-kernel real-time, multi-tasking and other characteristics.


FIG. 2.5 NB-IOT module implements


3.1 Intelligent Control and gateway system hardware overall:

Figure 3.1 intelligent control and overall system hardware gateway

3.2 Guangdong embedded development board Gateway 6818 display interface:

FIG Guangdong 3.2 gateway boards fitted display interface 6818

3.3 two terminal nodes

3.3 Set two end nodes of FIG.

3.4 NB-IOT module coordinator

FIG collector 3.4 with the coordinator module NB-IOT

3.5 gateway on the display panel module 1 offline

Figure 3.5 gateway module on the display panel 1 offline

3.6 TCP client display control platform

Figure 3.6 TCP client display control platform


3.7 web client data platform

Figure 3.7 internet web client data

3.8 achieved by mobile switching modules communicate with other NB-IOT

Figure 3.8 achieved by mobile switching the communications module NB-IOT

Curriculum design summary or conclusion

1 gateway importance of things:

Compared to the age of the Internet, networking communication protocols more diverse, fragmented matter is very serious, thus highlighting the importance of the gateway will – Things Gateway able to integrate different information was collected, and it transmission to the next level, and thus are unable to transmit information between the various parts. Things aware network gateway may be implemented with a communication network, and a protocol conversion between different types of network-aware; both wide area network, a local area network can also be implemented.


2 JPEG images fixed display problems:

6818 Gateway panel has an LCD screen 800 * 480, how to display images on the board, you can use the jpeg library. Its effect is to show the need for real-time sensing layer data, if you want to achieve good results is not directly replace the whole picture, which will involve replacing fixed images, to achieve the replacement of the background image and display real-time data.

Alternatively the whole picture of the original reference function modification function is void jpeg_show (int xx, int yy, char * fullfilename), a function of carrying X and Y axes and the offset address of the picture, the pixels by writing function lcd_draw_point (cinfo.output_scanline + yy, i + xx, color); write achieve site plus the offset image, the display effect designated replacement.


3 OpenCPU association:

Three years of experience with the development board can be considered a lot of it, ESP8266, ESP32, NB-IOT AT commands are supported by the development and OPenCPU class, why have OpenCPU it, and move away before the technician on the phone he gave me the reason is explained BC26 support OpenCPU is aimed at those who are reluctant to buy an external MCU used generally are applied using an external aT MCU development. Haha, a little here access is not on words.

According to personal development experience, AT commands easy to test, OPenCPU class easy to develop. If you have a built-IO port of the microcontroller is to achieve some specific function or simply forward data, but also data analysis carried out in strict accordance with the format of AT commands, it would be a difficult and their own perception of things.

As things engineering students, in the face of system development should be the proper use and development of AT OpenCPU class.


Programmers colleagues were contractors pit, and this society is very magical

No matter what life gives you kind of hit, we have to live hard.

Because live for the moment every minute, it will not let the name of the game, thinking about tomorrow’s work just fine.

Anyway, as long as the life beat to death you win.

On Friday went to the customer site, sitting room debugging debugging products, had such a conversation during which I thought for a long time, I give you a brief summary of the next story:
    1, responsible for their own home renovation contractors, first met in the course of conversation has said he is not easy, there are two there at four old son to raise. After being touched I chose them.
    2, there are a lot of problems during the renovation, are reluctant to point out after rectification, or give you a clear-cut, in short, feel uncomfortable.
    3, when the end of the paragraph, contractors trick, customers feel is not easy, conscience make life difficult, then settle the balance due.
    4, not long after the renovation, discover off the ceiling, requiring rework contractors, contractors over the past few face, swore and resentment. Customers can not only find the decoration company, the last renovation company to deposit the grounds of detention, contractors agreed to rework.
    5, customers can not figure out why I treat you so good, so you understand, but you do this to me.

First of all, in my opinion, the two sides are not wrong. Just not very clever, two of the world need to work for some time, but this time, because the two sides have different perceptions and values ​​had some differences.

Client was a mother, work is to write code in the unit, is very simple, two first-line life. Small contractors work outside the home, teens mixed society, slope to climb all the way Gunda broke into contact with team projects do today.

Contractors purpose is very pure, it is to make money. Below him so many people to feed, now a Shanghai carpentry, masonry by 500+ day is to count (Do not say you want to move bricks .. That you really do not necessarily suffer from eating), he was just money his way.

From our point of view, his “moral turpitude”, “conscience is eaten x”, “good or bad ah” and so on, but, uh, no one thing is illegal. He is doing one thing, “sets of information poor.”

If you know renovated trekking round after round, contractors buy you material they covered not you, decoration company can not make you money, and with contractors and hardware store owner also could not send your money.

It is because we do not understand, so it is necessary to find a professional people to do things, professional people reflected in this matter out of his expertise, incidentally clip brought you “contraband.” And we want people to Jiexin they often help us to do things that do not go wrong.

That from another perspective, when you write code, is not it 8 hours a day (996,007 I sympathize with you ..) were writing it? workload up to 50% yet?

We brought the issue of wages owner /, is to complete eight hours of work, the boss know that nothing if the workload can “rap” You have no due diligence, deducted from your salary, you add KPI. This time we will not blame the boss, “I’m dedicated to you to do things, I want no more, do not fight do not rob, why do you have to deduct my wages.”

After such an empathy, it is not a good heart Shou Lexie. As for the woman later said “we are too honest, and do not understand anything, too good conscience, deceived by their lying to the” noncommittal ……

Society is the real brushes, you can brush bright brush clean. Society is a big vat, you can dyed green, stained. Why do some people brush bright, still stained? Forced.
    There is a saying not often mention it: No one’s life is easy, if you think it is, it must be someone for you before loading the line.

I was such a person, but I never went to complain, because complaining is useless, ah, no one would give me the money, no one will help me. why? Because who knows only to help bury a all day long complained that this injustice, but they did not thought about how to change people?

Waiting for someone to feed it? Nor is it impossible, but someone feeds you can taste it? Full control pretty good. If you want to eat good, you do not have to rely on their own what.
    Later, the experience made me understand that the “altruism” is a how interesting things. Zhang brother, Yong Ge take in personal experience tells altruistic good.

In order to repay the reader, readers bring real benefits, I got some friends to some internal tickets, buy out of that. Last week, I took a few original 699 “IBM AI camp” tickets gave them a few readers took part. After the meeting readers sent me a long segment of feedback, I’m really happy, really can begin to help others, and screenshots have been authorized.

Anyway, so seriously, carefully readers, I believe that one day you can achieve what you want.

Here you can not help but to add another point: some eyes open, do not stared at the money.

Why say this thing, because the time before I give the reader with benefits, to secure some tickets for manufacturers of the General Assembly, but also to win some high temperature subsidy, readers are trying to sign up. The things I’ve done several months, attended the reader can set percentage of the population.

However, recent events make me a little …… something very simple, I made welfare paste, you sign up. One reader would add my micro letter, I specifically asked how much money, how to get. I say the end of the sponsors gave me, I gave you, my regular readers know everything. …… I was actually asked how I believe you want to give me ……

Before the change might hate go up, “Aisin letter, do not believe left, back of the queue of people waiting ……” But now it is changed a lot, and thank those who usually pointing or two of my friends. I will reply to the reader who “not bad at this point …… I promised a lot of points, you can send the screenshot the group” and then made a few envelopes map, and that makes him believe me.

I say these red envelopes are sent out …… I worse you what this 100 ……

This incident shows that Shane? They are a programmer, since concern me, I wanted to give you clear and clear, do not try to front that money, how much I did not. Do you think you earn ten thousand pieces a month, all the good 2,3w, and 100 can not buy a disadvantage not.

But you have to think so, it is easy to fall into the lump in the corner. To do one thing, your mind will become: “? How much” – “can not but ask for money, or cheaper,” – “I see,” in fact, time is wasted on little cost, real time cost with no regard.

If you look at things in the immediate, I can not imagine how the next five years to grasp a chance. Often years after the explosion chance, today is a loss, or do not make money. If the current is fast money back, and that in most cases, will not be particularly large room for growth.

Well, today a little too much nonsense, give us summarize:
    1, to get used to social phenomena, to learn to self-promotion.
    2, do altruistic things, feel good at the same time, might even have unexpected results.
    3, eyes open more, do not just stare in front of that money.

No public concern “Python Corner”, reply back “machine learning e-books,” get 100 free e-books related to machine learning oh ~


Storm Series (two) – Storm explain core concepts

A, Storm core concepts

1.1 Topologies (topology)

A full stream handler is called Storm Storm Topology (topology). It is one that is connected by Stream by the Spouts and Bolts directed acyclic graph, Storm will keep each submit to the topology of the cluster continues to run, in order to deal with a steady stream of traffic, until you take the initiative to kill (kill ) so far.

1.2 Streams (stream)

Stream is the core concept in the Storm. A Stream is an unbounded, and parallel processing in a distributed manner to create a Tuple sequence. Tuple may contain the most basic type, and data type definitions. In simple terms, Tuple is the actual carrier stream data, and Stream is a series of Tuple.

1.3 Spouts

Spouts is the source of the data stream, a Spout can send data to more than one of Streams. Spout reliable and unreliable usually divided into two types: a reliable Spout can resend Tuple in case of failure, unreliable Spout once the Tuple sent to ignore the.

1.4 Bolts

Bolts is a processing unit stream data, it can receive data from one or more of Streams, the processing is completed and then transmitted to the new Streams. Bolts may be performed by filtration (Filtering), the polymerization (Aggregations), connected (joins) and other operations, and can interact with the file system or database.

1.5 Stream groupings (group policy)

spouts and bolts on the cluster perform tasks are executed in parallel (as shown above, each circle represents a Task) by a plurality of Task. When a Tuple needs to be sent to Bolt B executed from Bolt A, to know how the program should be sent to Bolt B in which a Task enforce it?

This is caused by Stream groupings grouping strategy to decide, Storm follows in a total of eight built-in Stream Grouping. Of course, you can also implement custom Stream grouping strategy by implementing CustomStreamGrouping interface.

  1. Shuffle grouping

    Tuples randomly distributed to each Task each Bolt, each Bolt to get the same amount of Tuples.

  2. Fields grouping

    Streams are grouped by field (field) grouping specified. Tuples partition assumed by user-id field, the user-id of the same will be sent to the same Task.

  3. Partial Key grouping

    Streams are grouped by field (field) grouping specified, and Fields Grouping similar. However, the load balancing for two Bolt downstream, it can be better optimized in case the input data is not averaged.

  4. All grouping

    Streams will be replicated all of Bolt’s Tasks. Because of duplication of data processing, it is necessary to use caution.

  5. Global grouping

    Streams whole Bolt will enter one of the Task, usually into the id smallest Task.

  6. None grouping

    None grouping and Shuffle grouping current equivalent, they are randomly distributed.

  7. Direct grouping

    Direct grouping can only be used to direct streams. This way you need to specify Tuple directly from producers which are processed by Task.

  8. Local or shuffle grouping

    If there Tasks and objectives Bolt Bolt’s current Worker Tasks are in the same process, then the priority will Tuple Shuffled to target Bolt is on the Tasks of the same process, so you can minimize network traffic. Otherwise, consistent and common Shuffle Grouping behavior.

Two, Storm architecture Detailed

2.1 Nimbus process

Also known as Master Node, it is the overall commander Storm cluster work. The main functions are as follows:

    Thrift via the interface, and receives Topology Client monitor submitted;

    According to cluster resources Workers will Topology Client submitted assignments, assign the result is written Zookeeper;

    By Thrift interface Supervisor download request listening Topology code, and available for download;

    By Thrift interface to monitor UI reading of the statistics, read the statistics from the Zookeeper, returned to the UI;

    If after the process exits, immediately restart the machine, cluster operation is not affected.

2.2 Supervisor process

Also known as Worker Node, Storm cluster resource manager, started on demand Worker process. The main functions are as follows:

    The timing for new Topology code is not downloaded to the local, and regularly remove the old code from a Zookeeper Topology checks;

    According to Nimbus task allocation plan, needed to start one or more of the Worker processes in the machine, and monitor all of the Worker process conditions;

    If the process exits, immediately restart the machine, cluster operation is not affected.

2.3 zookeeper’s role

Nimbus and Supervisor course is designed for fast failure (the process of self-destruction encounter any unforeseen circumstances) and non-state (all states stored on disk or Zookeeper). The benefits of this design is that if their process is accidentally destroyed, after a reboot, you simply state before data can be acquired from the Zookeeper, and will not cause any data loss.

2.4 Worker Process

Task constructors Storm cluster, Task instance constructor Spoult or Bolt, start the Executor thread. The main functions are as follows:

    Launch one or more Executor threads in this process according to the task assigned on Zookeeper, and give the constructed Task instance to Executor to run;

    Writes a heartbeat to the Zookeeper;

    Maintaining the transmission queue is transmitted to the other of the Worker Tuple;

    If the process exits, immediately restart the machine, cluster operation is not affected.

2.5 Executor thread

Mandate holders Storm cluster loop executes Task code. The main functions are as follows:

    Performing one or more of the Task;

    Acker execution mechanisms responsible for sending Task processing status to the corresponding worker Spout is located.

2.6 parallelism

1 Worker process execution is a subset of the Topology of a Worker multiple Topology service will not happen, so a running Topology is composed of a plurality Worker processes in a cluster on multiple physical machines consisting of . 1 Worker process starts one or more threads to execute an Executor Topology of Component (components, namely Spout or Bolt).

Executor is a process to be launched Worker separate thread. Each Executor will run a Component in one or more of Task.

Task Component of the code units. Topology After starting, the number of Task Component 1 is fixed, but the number of threads that Component Executor used can be dynamically adjusted (e.g.: an Executor thread can execute one or more instances of the Task Component’s). This means that for a Component is, # threads <= #="" tasks="" (the="" number="" of="" threads="" less="" Task)="" this="" situation="" is="" present.="" The="" default="" Task="" Executor="" case="" equal="" to="" the="" that="" an="" run="" only="" one="" Task.<="" p=""/>

Summarized as follows:

    A running Topology by a cluster consisting of multiple Worker processes;

    By default, each Worker process starts a default Executor thread;

    By default, each Executor default Task start a thread;

    Task Component of the code units.

Reference material

  1. storm documentation -> Concepts

  2. Internal Working of Apache Storm
  3. Understanding the Parallelism of a Storm Topology
  4. Storm nimbus single node goes down processing

More big data series can be found GitHub open source project: Big Data Getting Started


Why did you become a programmer?

Brothers and sisters, remember himself as a programmer of early heart? Back Gongjin year, no, I think back to that year, it seems to be “destined” to go down this road. Well because it is not in the plan, so it takes a lot of curved road.

The road is long Come, we have to search up and down a bit, right when it is a recovery disk. I first said channel bits and pieces of their own programmers in this road, may be able to give you some inspiration or depressed and confused perception of. In different stages of life, we all need and their own past to say “bye” carefully before thinking about the future direction forward.

01, with a lump of TV

I remember four high (repeat) when the same table United Arab often laughed at me, said: “Computer not know, really admire you, not that ‘! TV with a lump of thing’,” Arab said the “knots”, refers to the keyboard. Indeed, ah, then summoned up the courage computers and TVs, very heavy, really is more than just a keyboard (not, as well as a host).

At that time most of the computer doing one thing to download movies in MP4. Then take advantage of the time to study hall at night, put on headphones secretly watching. Because the class teacher to accompany the lesson of the day more tired, often sat on the podium, do not know what they are doing, anyway, no energy at us hold up.

Then little-known Tang Wei starred in a movie very fast hardware – “Lust, Caution” (over, to the exposure age), we are regarded as being the male students. It was also ill to recommend to the several female classmates, and so they are also my MP4, I glanced out her face from a slight expression of joy, even though they had tried to cover up.

Repeat for a year, college entrance examination results are still unsatisfactory. United Arab recommended to me a few schools, I casually fill volunteer, and subject to the swap (why I was destined to become a programmer, is derived from this). He is the third year the repeat, apply a little more experience than I do, but the results are worse than I am. The spate of repetition, his attitude has been a collapse, when still in high school, “Hongzhi Ban” it. “Hongzhi Ban”, you know it, it can all learn Pa ah, but repeat again and again to destroy the confidence of the Arab League, forced him to become like me to learn Java.

Here, I must warn you about the young Benedictine College Shanghai Campus, the mentality is not strong enough Do not repeat, perhaps a little earlier into the community is a better choice. United Arab now stay at home, half a year to go to the wedding Pingdingshan when passing to his home to his chair, he even asked me eager to learn programming hard to learn, I want to teach him. I was shocked, OK.

We’re a tight, despite years never met, but still quite a young man talking to a high-spirited. Even chatted again with “Lust, Caution.”

02, computer network professional

College entrance examinations is a professional college of “Power Engineering” – United Arab recommended, the school is said to be the most popular professional (no one), many children who crush to get into the profession.

Fortunately, I am not the children, not naturally get on such a good professional. I was to swap the “computer network” professional, the worst professional school (no one). It is said that the predecessor of the school was founded in 1933, concedes, but “computer network” professional only less than two years of age (us this session is the second year), regarded as a birth right of a small doll.

At that time, only that “computer network” is the worst professional, it is also one of the worst school students. We specialize in more than a hundred students, almost no one had full confidence in the future. Everyone’s mind is the heart, through the efforts of half a semester a semester to go to the electric power system. Even turn-based hopeless, after graduation did not intend to become a programmer eight children.

I have two years of college, a 破罐子破摔 appearance. Class with a laptop to play games (stand-alone, such as Teenage Mutant Ninja Turtles, Dragon, Need for Speed, etc.), class time is still playing. Not willing to spend a little time to buy food, they are places of Friends Bangdai. Eat play games, this depends on a small table below.

Now go back to standing height, I really want to hammer the abandonment of their own at that time.

Due to writing, know a lot of young (98 years later), freshman to start writing. In addition to writing, but also engage in public numbers, sell courses, and so on, all kinds of knowledge are paid to play BHC ah.

If I had a time machine, I would have started writing a freshman, then toss some open source projects. You may want to smoke I could not help but have a few Erguang Zi – Wake up, brother, still dreaming?

Unfortunately, I do not have a time machine. But I want young people to lead me to quit ah, especially the blessing of the body I do not know (I think professional is very bad, in fact, very fast hardware), do not easily give up on themselves ah in happiness. We looked down on the year by professional computer network, especially learning the Java programming language, in the first-tier cities, but very popular ah.

Unfortunately, that time teachers did not give us any advice (they also feel themselves from the school’s attention, no hope of promotion), it also encourages us to turn to the special power professional. And we are fools, in addition to playing all day tour game, leaving nothing up.

Btw, when students turn to the power line, now mostly working in a small city Electricity Authority, promotion basically hopeless, because other qualifications older retirement or career change ah.

03, we do not train

Unexamined spent two years of college life, no hope of employment after graduation conscious, and more than two dozen students on the south to Suzhou, a trained software park ( Two months later, he got a job, a month starting salary is only 1,200 yuan, 800 yuan in addition also training fee per month, the rest of that money barely able to make ends meet.

Looking back now, it was completely unnecessary to go to training. After all, we learned two years of Java programming, SQL Server also learned, the underlying computer network architecture also learned, pages Musketeers is also a beginning, fully capable to first-tier cities to find a job.

Unfortunately, that time relatively ignorant that he himself looked down upon. I know some large Internet cafes, the early years are desperate to go to a computer professional from another professional. This is the gap ah!

If that time can encounter such a person like me now, pointing to the maze, even without pointing, just say a word of encouragement: “Young man, go north drift it, go deep drift of it, the future belongs to you.” that life may be completely different.

Cognitive really too important. If you understand the current economic environment, industry environment, industry environment, technical environment, you can make better decisions.

Of course, after all, still I have to thank about inclusive IT industry! As the saying goes: “white black cat, catch mice is a good cat.” Those technical education, promising abound; those training background, dry out some big business are also many.

If you feel trapped and training should be regarded as a very wise choice, it allows you to battle the state quickly, in actual combat learning, progress, and then become battle-hardened after a towering tree. In the Three Kingdoms era, the “Riding Alone for Thousands of Miles” Guan Gong, in the era of liberation, there Peng General “dare Hengdao immediately,” the. In short, a hero abound in actual combat.

If your self-learning ability is relatively strong, and there is no need to go to training, learn programming information on the network has a lot, I can offer some to you.

04, finally

I became a programmer, put it nicely is because the “fate” did not sound good point is the “cock.” How about you?

A: Because the desperate B: I do not know why C: Because poor D: Because the fate E: because I like to knock Code F: because the pursuit of high-paying G: Because you want to change the world

But no matter what the early heart, I want you to “programmer” pride, and only then can you go more determined, more long-term.



Continuous Integration with Jenkins ssh Jenkins cli advanced part of the

Series catalog

Jenkins Cli Introduction

A cli tool to Jenkins Jenkins Cli provided by this tool is very powerful, can be done such as restarting jenkins, create / delete the job, view job console output, add / delete nodes and other functions, but the actual work, such as job creation configuration is clear cli very difficult, as directly in the web management interface operation, but more convenient for the restart Jenkins, view diagnostic information, perform a manual build tasks, etc., is used directly cli than entering the web management interface operation. Therefore, when the web management interface when to use cli, depends on whether there is conducive to enhancing productivity, whether it helps enhance their ability, whether it helps to enhance the team’s automation levels of these indicators do not think the use of high levels cli on behalf of the blind use cli leading to reduced efficiency or increasing problem.

jenkins jenkins cli can jar package provided by the environment or to create a cli cli performed using the ssh client, this section will also introduce them separately.

Cli use to perform their own client jenkins

We opened jenkins installation directory, go to war \ WEB-INF directory, this directory has a jenkins-cli.jar file, which is used to create jenkins cli environment. We open a command-line tool in this directory, then execute java -jar jenkins-cli.jar help will be able to see its output help information

You can see the -s option to specify a url, the url is the url jenkins web management interface, is now testing environment, url is http: // localhost: 8080, of course, want to perform cli, also you need to enter your user name and password here specify the user name and password by -auth option.

We put the information together to form the following command

java -jar jenkins-cli.jar -s http://localhost:8080 -auth tylerzhou:密码 help

The above statement, help for the command to be executed.

Note that the command executed here is the direct command name can not be added as other command-line tools – – and / or logo

If you can not find Jenkins-cli.jar, can enter in the browser http: // localhost: 8080 / jnlpJars / jenkins-cli.jar download save, pay attention to the above address is replaced with your actual address

Click Enter, you can see a brief description of all Jenkins cli executable commands and commands. Here are a few may be more commonly used commands

1) list-jobs, Jenkins can be listed in all of the job, like opening jenkins web management interface Home seen.

The following command

java -jar jenkins-cli.jar -s http://localhost:8080 -auth 您的账户:您的密码 list-jobs

Note that, when Jenkins cli command on every execution need to bring java -jar jenkins-cli.jar -s http: // localhost: 8080 -auth your account: Your password + command to be executed, and enter the command only after entering an environment not on the line.

2) build + To build the job name can be triggered by a build cli.

After building a successful, we open the web management interface can see more than one building.

cli command building is a very useful command, although most of our tasks are based on an automated process git, but sometimes also need to perform some manual script automation to accomplish some work in. If every time you open web management interface obviously not as good as executed manually to perform high-efficiency cli.

3) restart / safe-restart service can restart jenkins, you can see the safe-restart is safe to restart by their names, it will wait until all operations are completed and then carries out a restart

4) clear-queue Clear build queue. Jenkins no practical experience with friends may not know their own reasons or due Jenkins bug our test script to inadequate, and sometimes can cause obstruction build Jenkins, has been in a state unable to complete the building, which when the web administration interface click cancel to build can not be canceled at this time in an attempt to stay the order. Of course, it may still not be terminated at this time need to use violence restart command to restart the service.

Use api token landing

We are all above username: password to log in the way of cli, so that the clear-text passwords exposed is not desirable, in fact, Jenkins cli can also Username: to achieve landing cli apitoken Here we introduce five ways how to set api. token.

We open jenkins web management interface, click on the button to the left of the People, the interface will appear in the list may be more than one user, then click on the user’s own login name jenkins, click the configure button to enter the interface, the interface appears find Api token column

Click Add new token generates token there will be a small screen, enter the name of the token, which can be any name, then click generate, this time it will generate a token

Note that this token must take down the copy and save it somewhere else, next time to find open when he can not see it.

Click Save when done.

At this point we enter the following command in cli

java -jar jenkins-cli.jar -s http://localhost:8080 -auth tylerzhou:11f52cef1324556a41d966083ffcf0ac1b

Which tylerhzou user name, is behind the token we just created

If successful, the output of the above command on all of jenkins cli command.

Command information saved to a separate file

We can see that the way to use the token landing will lead to the command line is very long, seriously affect the readability of command, in fact, we can save the information to a file auth file. I Jenkins installation directory / war / WEB-INF directory create a text file called password.txt, as follows


That username above, enter the auth: apitoken

Then we specify a username (password or token) file by specifying the file name @ -auth option in the way to achieve land.

code show as below

java -jar jenkins-cli.jar -s http://localhost:8080 -auth @password.txt

Above the back of the @ password.txt is the password file we created

SSH to log in Jenkins execute cli

Use SSH to log in Jenkins needs to be configured, by default is a random port number jenkins SSHD port to use, this is clearly not conducive to ssh login using ssh landing we need to explicitly know the port number is how much we open the Manage jenkins> Configure Global Security find SSH Server field, the default random option to cut fixed and enter a different port number, I am using the 16022 click save configuration. port configured, we also need to add the public key to the current user profile entry, refer to a landing above api token, enter the current user interface, this time we do not configure apitoken, but pulled down to find the SSH Public Keys option, the copy we generate ssh public key here. see About generate ssh key other information about the previous chapters, not repeat them here.

After completion, we can execute commands via ssh way.

Enter the following command

ssh -l tylerzhou -p 16022 localhost help

-L which specified user account when we landed jenkins.
    If the configuration is successful, the above command will list all the commands and a brief description of Jenkins.

Ssh command executed by the above command is executed jenkins-cli.jar like can refer to each other.

Simple encapsulation of jenkins cli

Using simple packaging bat

Can be seen, whether by performing the above cli tool or ssh, always need to bring some fixed login information, very tired, we can make a simple package, so that each time only need to enter the command, no longer need each re-enter fixed content

We put it into a bat encapsulated command, the command is

@echo off
set /p arg="请输入您的命令: "
java -jar jenkins-cli.jar -s http://localhost:8080 -auth @password.txt %arg%
goto start

The above command to create a start producing first label, and then prompts the user to input commands, input later spread to the jenkins cli tool, and then execute the goto statement to jump start the label. Repeat the above command, you want to terminate need to press ctrl + c to termination

Using powershell script package

As the author of the bat is not very skilled, and therefore can write a complex script feel more strenuous, there were about using powershell package to support clear screen to view the execution status and exit options, the script reads as follows

  "  退出请按1或者输入exit
  Write-Host $writeout

while ($true) {
$myvar=Read-Host "请输入命令 "
if(($myvar -eq 1)-or($myvar -eq "exit")){break}
elseif (($myval -eq 2) -or ($myvar -eq "cls") -or($myvar -eq "clear")) {
elseif ($myvar -eq 3) {
    if($LASTEXITCODE -eq 0){
      Write-Host "执行成功"
        Write-Host "执行失败"
elseif($myvar -eq 4){
Write-Host $writeout
else {
   java -jar jenkins-cli.jar -s http://localhost:8080 -ssh -user tylerzhou $myvar.split(" ")

We create a ps1 file, copy the above into account and follow the instructions, you can execute the script.

windows no problem ssh client

Some children’s shoes when doing follow the above way of landing ssh, ssh may be entered at the command line prompt command not found, because only the latest version of windows 10 (not all versions have). Fortunately, Jenkins-cli .jar also provides ssh landing way. we enter the following command

java -jar jenkins-cli.jar -s http://localhost:8080 -ssh -user tylerzhou help

Ssh designated landing approach in the above manner, and specify the -user, do not need to specify the port, you can use ssh way connected. We can simply look at using closed closed method mentioned above, so you do not have to keep entering repetitive, fixed content.


Learn concurrent programming, a thorough understanding of these three core is the key


An article on the concurrency into the world, do not miss to everyone with a concurrent programming appetizers, then we gradually over dinner, before eating dinner, I would like to quote those verses: “Wang Ling side, near and far different level, “far to see the profile in recent look at the details, constantly switching or thinking perspective to learning

From a distance, concurrency, concurrent programming can be abstracted into three core issues: the division of labor, sync / collaboration, mutual exclusion

If you have been working, then you must have heard or are applying agile development model to deliver daily tasks, we use to explain the process you are familiar with the three core issues

Division of work

The current split into Sprint’s Story “appropriate” size of the Task, and arrange to “appropriate” to complete the Team Member

There used two “proper” Story split into moderate size, to be completed by the Task is very important. Split too coarse, leading to the difficulty of this task is completed is high, time-consuming and difficult to work with others; split size is too small, has led to too many tasks, poor management and tracking, waste of energy and resources. (Suitable thread in order to better complete piece of work, of course, a thread can easily get no need multi-threading); arrangements to the appropriate personnel to accomplish equally important, UX-UE issues to the back-end processing staff, apparently there is a problem (the main thread is supposed to do to the child thread is obviously not solve the problem, each thread is doing the right thing can play a role)

Division of Labor, the common Executor, producer – consumer model, Fork / Join, etc., which are reflected in the division of thought

Sync / Collaboration

Task split is completed, I have to wait for the seating of the tasks, such as John Doe Joe Smith to task, that is to say there is a dependency between tasks, completed tasks to perform in front of, behind the task can only be performed by people in senior communication repeatedly confirmed, to ensure that their tasks can be started. But the face of the program, we need to understand the communication process, a thread executing the task, how to notify subsequent thread execution

All sync / collaboration we can use you are most familiar with If-then-else to represent:


The above code is to say: when a certain condition is not met, the thread needs to wait; when a certain condition is met, the thread needs to be awakened to perform, the collaboration between the threads may be in collaboration with the main thread child thread, it may be sub-thread cooperation with sub-threads, Java SDK in CountDownLatch and CyclicBarrier thread is used to solve the problem of collaboration


Division of labor and synchronization emphasized that performance, but exclusive emphasis on correctness, is what we often referred to “thread-safe” when multiple threads access a shared variable / member variables, uncertainty can occur, resulting in no the main uncertainty is visible, atomicity, ordering these three issues, the core of solving these problems is mutually exclusive


The same time, allowing only one thread to access shared variables

Look at the chart, the trunk road is shared variables into the main road can only have a car, so if you understand it? “World affairs, long period of division.”

The same Java SDK, there are many mutually exclusive solutions, for example, you’ll be able to think of the synchronized keyword, Lock, ThreadLocal so is mutually exclusive solutions

to sum up

Crazy capitalists extract surplus value of labor workers, to get maximum benefit. When you are faced with the CPU, memory, IO these labor workers, you are the capitalists, you need to think about how to best extract their value

When a worker capable of living, never let two people to dry (not necessary in order to meet the single-threaded multi-threaded)
    When multiple workers to work, we should let them clear division of labor, cooperation and smooth, no contradiction

When the task is large, due to the slow work IO, CPU work fast, there is no need to let the current IO CPU die, etc., turn to execute other instructions, this is the surplus value, how to maximize the extraction of its value, which it comes to subsequent tuning issues, such as how many threads and other appropriate

The division is to design, synchronization and mutual exclusion is achieved, there is no good without a good design to achieve, so in the division stage, it is strongly recommended that you outline sketch, to understand where the bottlenecks are, as this will better achieve subsequent chapters content, I will lead the sketches, analyze problems, and gradually develop the habit

In this chapter you can simply use the following chart summarizes the contents of leaf nodes we will gradually light up, at this stage without too much attention (if you chew up on the JDK source, you might lose the pain, and eventually give up your Advanced the road)

Understanding the three core issues, you have to be fully integrated real life, concurrency problems in the program, basically can find the prototype in real life

Contents next article, we will talk about three issues caused by thread-safe: “Visibility, atomicity, ordering”, which involves a little content JMM, you can find out in advance so that we can better collision

Soul questioning

    Work in multithreaded programming scene more than you?

    Think of multi-threading, synchronized only think of it?

    Java and contracting each category, you have to understand the underlying implementation and design it?

Improve productivity tools

Recommended Reading

    Every day SpringBoot, still do not understand RESTful API returns a unified data format is how to achieve?

    Parents delegation model: manufacturers of high-frequency interview questions, easy to get

    Interview not know the difference between BeanFactory and the ApplicationContext?

    How to design a good RESTful API

    Red-black tree, super movement chart Detailed, easy to understand

Welcome to continued public concern number: “arch day a soldier.”

    Dry cutting-edge Java technology sharing

    Productivity tools summary | Reply “Tools”

    Analysis of interview questions and answers

    Technical Data collection | Reply “Information”

Easy to read detective novels thinking fun learning Java technology stack-related knowledge, in line with the simplification of complex issues, problems specific abstract and graphical decomposition of the principle of progressive technical issues, technology continues to update, please stay tuned ……