Categories
Uncategorized

Shiro achieve dynamic resource for fine-grained user permissions check

Foreword

In actual system applications, such a common business scenarios, the need to achieve user access to resources to be dynamically permission check.
    For example, in a business system platform, there is a business, brand, product and other business resources. The relationship between them is: a merchant can have multiple brands under one brand can have multiple items.

A business user can have multiple accounts, each account has different levels of authority.
    For example, Wang is responsible for operating the work of all the resources of the business under A, Xiao Zhang is responsible for operating the work of all goods under the brand A and brand A. And Mike is responsible for the brand B

Shiro itself provides RequiresAuthentication, RequiresPermissions and RequiresRoles and other annotations are used to achieve static certification authority,
    But not suitable for dynamic resources such fine-grained permissions certified check. Based on the above description, the article is to add a check for fine-grained access to dynamic resources.

Probably design ideas

    1. Add a custom annotation Permitable, for the conversion of resources into a string representing the rights shiro (support SpEL expression)

    2. Add a new AOP section, used to associate a custom method of tagging and annotation Shiro permissions check

    3. check whether the current user has sufficient privileges to access protected resources

Coding

    1, the new PermissionResolver Interface

import java.util.Collections;
import java.util.List;
import java.util.Optional;

import static java.util.stream.Collectors.toList;

/**
 * 资源权限解析器
 *
 * @author wuyue
 * @since 1.0, 2019-09-07
 */
public interface PermissionResolver {

    /**
     * 解析资源
     *
     * @return 资源的权限表示字符串
     */
    String resolve();

    /**
     * 批量解析资源
     */
    static List resolve(List list) {
        return Optional.ofNullable(list).map(obj -> obj.stream().map(PermissionResolver::resolve).collect(toList()))
                .orElse(Collections.emptyList());
    }

}

    2, the new entity class business resources, and implement PermissionResolver interfaces, where the trade resources, for example, such as New Product.java

import com.wuyue.shiro.shiro.PermissionResolver;
import lombok.Getter;
import lombok.Setter;
import lombok.ToString;
import org.hibernate.annotations.GenericGenerator;

import javax.persistence.*;
import java.util.Date;

@Getter
@Setter
@ToString
@Entity
@Table(name = "product")
public class Product implements PermissionResolver {

    @Override
    public String resolve() {
        return merchantId + ":" + brandId + ":" + id;
    }

    @Id
    @GenericGenerator(name = "idGen", strategy = "uuid")
    @GeneratedValue(generator = "idGen")
    private String id;

    @Column(name = "merchant_id")
    private String merchantId;

    @Column(name = "brand_id")
    private String brandId;

    @Column(name = "name")
    private String name;

    @Column(name = "create_time")
    private Date createTime;

    @Column(name = "update_time")
    private Date updateTime;

}

    3, the new custom annotation Permitable

import java.lang.annotation.*;

/**
 * 自定义细粒度权限校验注解,配合SpEL表达式使用
 */
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface Permitable {

    /**
     * 前置校验资源权限表达式
     *
     * @return 资源的权限字符串表示(如“字节跳动”下的“抖音”可以表达为BYTE_DANCE:TIK_TOK)
     */
    String pre() default "";

    /**
     * 后置校验资源权限表达式
     *
     * @return
     */
    String post() default "";

}

    4, the new authority checking section

import lombok.extern.slf4j.Slf4j;
import org.aopalliance.intercept.MethodInterceptor;
import org.apache.commons.lang3.StringUtils;
import org.apache.shiro.SecurityUtils;
import org.springframework.aop.support.StaticMethodMatcherPointcutAdvisor;
import org.springframework.context.expression.MethodBasedEvaluationContext;
import org.springframework.core.DefaultParameterNameDiscoverer;
import org.springframework.core.annotation.AnnotationUtils;
import org.springframework.expression.EvaluationContext;
import org.springframework.expression.spel.standard.SpelExpressionParser;

import java.lang.annotation.Annotation;
import java.lang.reflect.Method;
import java.util.List;

/**
 * 静态自定义权限认证切面
 */
@Slf4j
public class PermitAdvisor extends StaticMethodMatcherPointcutAdvisor {

    private static final Class[] AUTHZ_ANNOTATION_CLASSES =
            new Class[] {
                    Permitable.class
            };

    public PermitAdvisor(SpelExpressionParser parser) {
        // 构造一个通知,当方法上有加入Permitable注解时,会触发此通知执行权限校验
        MethodInterceptor advice = mi -> {
            Method method = mi.getMethod();
            Object targetObject = mi.getThis();
            Object[] args = mi.getArguments();
            Permitable permitable = method.getAnnotation(Permitable.class);
            // 前置权限认证
            checkPermission(parser, permitable.pre(), method, args, targetObject, null);
            Object proceed = mi.proceed();
            // 后置权限认证
            checkPermission(parser, permitable.post(), method, args, targetObject, proceed);
            return proceed;
        };
        setAdvice(advice);
    }

    /**
     * 匹配加了Permitable注解的方法,用于通知权限校验
     */
    @Override
    public boolean matches(Method method, Class targetClass) {
        Method m = method;

        if (isAuthzAnnotationPresent(m)) {
            return true;
        }
        return false;
    }

    private boolean isAuthzAnnotationPresent(Method method) {
        for (Class annClass : AUTHZ_ANNOTATION_CLASSES) {
            Annotation a = AnnotationUtils.findAnnotation(method, annClass);
            if ( a != null ) {
                return true;
            }
        }
        return false;
    }

    /**
     * 动态权限认证
     */
    private void checkPermission(SpelExpressionParser parser, String expr,
                                 Method method, Object[] args, Object target, Object result){

        if (StringUtils.isBlank(expr)){
            return;
        }

        // 解析SpEL表达式,获得资源的权限表示字符串
        Object resources = parser.parseExpression(expr)
                .getValue(createEvaluationContext(method, args, target, result), Object.class);

        // 调用Shiro进行权限校验
        if (resources instanceof String) {
            SecurityUtils.getSubject().checkPermission((String) resources);
        } else if (resources instanceof List){
            List list = (List) resources;
            list.stream().map(obj -> (String) obj).forEach(SecurityUtils.getSubject()::checkPermission);
        }
    }

    /**
     * 构造SpEL表达式上下文
     */
    private EvaluationContext createEvaluationContext(Method method, Object[] args, Object target, Object result) {
        MethodBasedEvaluationContext evaluationContext = new MethodBasedEvaluationContext(
                target, method, args, new DefaultParameterNameDiscoverer());
        evaluationContext.setVariable("result", result);
        try {
            evaluationContext.registerFunction("resolve", PermissionResolver.class.getMethod("resolve", List.class));
        } catch (NoSuchMethodException e) {
            log.error("Get method error:", e);
        }
        return evaluationContext;
    }

}

    5, realize authorized users

    /**
     * 授权
     */
    @Override
    protected AuthorizationInfo doGetAuthorizationInfo(PrincipalCollection principals) {
        Map principal = (Map) principals.getPrimaryPrincipal();
        String accountId = (String) principal.get("accountId");

        // 拥有的商家资源权限
        List merchantLinks = accountService.findMerchantLinks(accountId);
        Set merchantPermissions = merchantLinks.stream().map(AccountMerchantLink::getMerchantId).collect(toSet());
        SimpleAuthorizationInfo authzInfo = new SimpleAuthorizationInfo();
        authzInfo.addStringPermissions(merchantPermissions);

        // 拥有的品牌资源权限
        List brandLinks = accountService.findBrandLinks(accountId);
        Set brandPermissions = brandLinks.stream().map(link -> link.getMerchantId() + ":" + link.getBrandId()).collect(toSet());
        authzInfo.addStringPermissions(brandPermissions);

        return authzInfo;
    }
  • 6, custom annotation application

    6.1, access to business information based on the id

      @Permitable(pre = "#id")
      @Override
      public Optional findById(String id) {
          if (StringUtils.isBlank(id)) {
              return Optional.empty();
          }
          return merchantDao.findById(id);
      }

    6.2, obtain product information based on the id

      @Permitable(post = "#result?.get().resolve()")
      @Override
      public Optional findById(String id) {
          if (StringUtils.isBlank(id)) {
              return Optional.empty();
          }
          return productDao.findById(id);
      }

    6.3, find the list of goods under the brand

      @Permitable(post = "#resolve(#result)")
      @Override
      public List findByBrandId(String brandId) {
          if (StringUtils.isBlank(brandId)) {
              return Collections.emptyList();
          }
          return productDao.findByBrandId(brandId);
      }
  • 7, test

7.1, according to the service scenario described above, the user data to prepare 3

7.2, log on using the Wang test

7.2.1, obtain business information (have permission)

7.2.2, obtain product information (have permission)

7.3, log on using Li test

7.3.1, obtain business information (insufficient privileges)

7.3.2, obtain product information (insufficient privileges)

7.3.3, obtain product information (have permission)

7.4 Summary

As it can be seen from the above screenshot of the interface testing, in line with this program we designed at the beginning of business scenarios to be achieved.

Complete source code

Categories
Uncategorized

Allure-pytest features introduced

Foreword

Allure framework is a flexible lightweight multi-language test reporting tools, which not only shows the way to the web Introduction of test results, and allows everyone to participate in the development process to maximize the extraction of useful information from the tests performed daily from dev / qa point of view, Allure report simplifies common statistical flaw: the failure of the test can be divided into bug and interrupted test, you can also configure the log, step, fixture, accessories, timing, execution history and a TMS management and bUG system integration, so the above configuration, all the developers and testers responsible for testing the information can grasp as much as possible. From the point of view manager, it provides a clear Allure “big picture,” which includes the characteristics have been covered, the location of the defect aggregate appearance execution timeline and many other things easily. allure modularity and scalability ensures that you can always fine-tune some things to make Allure is right for you, so today we are concerned that how to make reports to display more detailed information we need, and the allure of jenkins integration

Generate reports

How to write pytest Framework Program to generate test reports, and will not be explained, the specific process may refer to: pytest advanced test report of html

Note: python plugin is used allure allure-pytest

Test code

In order that we can quickly recognize all the features of the allure, accompanied by a full test code

"""
------------------------------------
@Time : 2019/8/28 19:50
@Auth : linux超
@File : conftest.py
@IDE  : PyCharm
@Motto: Real warriors,dare to face the bleak warning,dare to face the incisive error!
@QQ   : [email protected]
@GROUP: 878565760
------------------------------------
"""
import pytest
import allure



@pytest.mark.hookwrapper
def pytest_runtest_makereport(item):
    outcome = yield
    report = outcome.get_result()
    report.nodeid = report.nodeid.encode("utf-8").decode("unicode_escape")  #

Solve the garbage

@allure.step("

Open your browser

") def fixture_step(): pass @pytest.fixture def init_url(): fixture_step() yield True

conftest.py

"""
------------------------------------
@Time : 2019/9/4 21:05
@Auth : linux超
@File : test_allure_feature.py
@IDE  : PyCharm
@Motto: Real warriors,dare to face the bleak warning,dare to face the incisive error!
@QQ   : [email protected]
@GROUP: 878565760
------------------------------------
"""
import pytest
import allure
import os


def login(username=None, password=None):
    """

Login simulation

""" user = "

Super linux

" pwd = "123456" if user == username and pwd == password: return {"code": 1001, "msg": "

login successful

", "data": None} elif "" == password or password is None and username: return {"code": 1002, "msg": "

password can not be blank

", "data": None} elif "" == username or username is None and password: return {"code": 1003, "msg": "

Username can not be empty

", "data": None} else: return {"code": 1004, "msg": "

wrong user name or password

", "data": None} @allure.step("

Enter your user name

") def input_username(user): print("

Enter your user name

") return user @allure.step("

enter password

") def input_password(pwd): print("

enter password

") return pwd login_success_data = [ #

Test Data

{ "case": "

The correct user name and password correctly

", "user": "

Super linux

", "pwd": "123456", "expected": {"code": 1001, "msg": "

login successful

", "data": None} } ] login_fail_data = [ { "case": "

The correct user name, password is empty

", "user": "

Super linux

", "pwd": "", "expected": {"code": 1002, "msg": "

password can not be blank

", "data": None} }, { "case": "

User name is blank, the password is correct

", "user": "", "pwd": "

linux super-brother

", "expected": {"code": 1003, "msg": "

Username can not be empty

", "data": None} }, { "case": "

User name wrong, wrong password

", "user": "linux", "pwd": "linux", "expected": {"code": 1004, "msg": "

wrong user name or password

", "data": None} } ] username_none = [ { "case": "

The default user name parameter

", "pwd": "123456", "expected": {"code": 1003, "msg": "

Username can not be empty

", "data": None} } ] password_none = [ { "case": "

The default password parameters

", "user": "

Super linux

", "expected": {"code": 1002, "msg": "

password can not be blank

", "data": None} } ] #

Change output

ids_login_success_data = [ "

Test} {username: password} {} {} {expectations

". format(data["case"], data["user"], data["pwd"], data["expected"]) for data in login_success_data ] ids_login_fail_data = [ "

Test} {username: password} {} {} {expectations

". format(data["case"], data["user"], data["pwd"], data["expected"]) for data in login_fail_data ] ids_username_none = [ "

{} {} Test password expectation {}

". format(data["case"], data["pwd"], data["expected"]) for data in username_none ] ids_password_none = [ "

Test {} Username: {} {} expectations

". format(data["case"], data["user"], data["expected"]) for data in password_none ] @allure.feature("

Login Module

") class TestLogin(object): @allure.severity(allure.severity_level.BLOCKER) @allure.story("

Login successful test

") @allure.title("

Login success scenario - {data}

") @pytest.mark.parametrize("data", login_success_data, ids=ids_login_success_data) def test_login_success(self, data): """

Login successful test

""" user = input_username(data["user"]) pwd = input_password(data["pwd"]) result = login(user, pwd) assert result == data["expected"] @allure.severity(allure.severity_level.CRITICAL) @allure.story("

Login failed test

") @pytest.mark.parametrize("data", login_fail_data, ids=ids_login_fail_data) def test_login_fail(self, data): """

Test user name or password is incorrect

""" user = input_username(data["user"]) pwd = input_password(data["pwd"]) result = login(user, pwd) assert result == data["expected"] @allure.severity(allure.severity_level.MINOR) @allure.story("

Testing the username parameter is missing

") @pytest.mark.parametrize("data", username_none, ids=ids_username_none) def test_username_none(self, data): """

The default user name test

""" pwd = input_password(data["pwd"]) result = login(password=pwd) assert result == data["expected"] @allure.severity(allure.severity_level.MINOR) @allure.story("

Password missing test parameters

") @pytest.mark.parametrize("data", password_none, ids=ids_password_none) def test_password_none(self, data): """

The default password test

""" user = input_username(data["user"]) result = login(username=user) assert result == data["expected"] @allure.severity(allure.severity_level.MINOR) @allure.story("

Test initialization address

") @allure.testcase("https://www.cnblogs.com/linuxchao/", "

Test address

") def test_init_url(self, init_url): flag = init_url assert flag is True @allure.severity(allure.severity_level.NORMAL) @allure.story("

Test fails use cases supplemented with Example attachments

") @allure.link("https://www.cnblogs.com/linuxchao/", name="

bug link

") @allure.description("

This is a test case has failed to perform

") def test_failed(self): """

You can also add a description of the use case here, but will be covered allure.description

""" try: assert False except AssertionError as e: with open("attach.png", "rb") as f: context = f.read() allure.attach(context, "

Pictures error

", attachment_type=allure.attachment_type.PNG) raise e @allure.severity(allure.severity_level.TRIVIAL) @allure.story("

Test cases broken by

") @allure.issue("https://www.cnblogs.com/linuxchao/", "

Broken links

") def test_broken(self): """broken""" with open("broken.json", "r", encoding='utf8') as f: f.read() @allure.severity(allure.severity_level.TRIVIAL) @allure.story("

Test unconditional skip test cases

") @pytest.mark.skip(reason="

Unconditional skip

") def test_skip(self): """skip""" pass if __name__ == '__main__': pytest.main(["-vsq", "--alluredir", "./allure-results", ]) os.system(r"allure generate --clean ./allure-results -o ./allure-report")

test_allure_feature.py

"""
------------------------------------
@Time : 2019/8/28 19:45
@Auth : linux超
@File : test_allure_fixture.py
@IDE  : PyCharm
@Motto: Real warriors,dare to face the bleak warning,dare to face the incisive error!
@QQ   : [email protected]
@GROUP: 878565760
------------------------------------
"""
import pytest
import os
import allure


def function_scope_step():
    print("function_scope_step")


def class_scope_step():
    print("class_scope_step")


def module_scope_step():
    print("module_scope_step")


def session_scope_step():
    print("session_scope_step")


def step_inside_test_body():
    print("step_inside_test_body")


@pytest.fixture(params=[True, False], ids=['param_true', 'param_false'])
def function_scope_fixture_with_finalizer(request):
    if request.param:
        print('True')
    else:
        print('False')

    def function_scope_finalizer():
        function_scope_step()

    request.addfinalizer(function_scope_finalizer)


@pytest.fixture(scope='class')
def class_scope_fixture_with_finalizer(request):
    def class_finalizer_fixture():
        class_scope_step()

    request.addfinalizer(class_finalizer_fixture)


@pytest.fixture(scope='module')
def module_scope_fixture_with_finalizer(request):
    def module_finalizer_fixture():
        module_scope_step()

    request.addfinalizer(module_finalizer_fixture)


@pytest.fixture(scope='session')
def session_scope_fixture_with_finalizer(request):
    def session_finalizer_fixture():
        session_scope_step()

    request.addfinalizer(session_finalizer_fixture)


@allure.severity(allure.severity_level.BLOCKER)
@allure.feature("

fixture scene

") class TestClass(object): def test_with_scoped_finalizers(self, function_scope_fixture_with_finalizer, class_scope_fixture_with_finalizer, module_scope_fixture_with_finalizer, session_scope_fixture_with_finalizer): step_inside_test_body() if __name__ == '__main__': pytest.main(["-vsq", "--alluredir", "./allure-results", ]) os.system(r"allure generate --clean ./allure-results -o ./allure-report")

test_allure_fixture.py

[
  {
    "name": "Ignored tests",
    "matchedStatuses": ["skipped"]
  },
  {
    "name": "Infrastructure problems",
    "matchedStatuses": ["broken", "failed"],
    "messageRegex": ".*bye-bye.*"
  },
  {
    "name": "Outdated tests",
    "matchedStatuses": ["broken"],
    "traceRegex": ".*FileNotFoundException.*"
  },
  {
    "name": "Product defects",
    "matchedStatuses": ["failed"]
  },
  {
    "name": "Test defects",
    "matchedStatuses": ["broken"]
  }
]

categories.json

Browser=Chrome
Browser.Version=63.0
Stand=Production
ApiUrl=127.0.0.1/login
python.Version=3.6

environment.properties

"""
------------------------------------
@Time : 2019/9/3 14:21
@Auth : linux超
@File : run.py
@IDE  : PyCharm
@Motto: Real warriors,dare to face the bleak warning,dare to face the incisive error!
@QQ   : [email protected]
@GROUP: 878565760
------------------------------------
"""
import pytest
import os


if __name__ == '__main__':
    pytest.main(["-sq",
                 "--alluredir", "./allure-results"])
    os.system(r"allure generate --clean allure-results -o allure-report")

run.py

Directory Structure

Allure properties

Environment

Add Allure environmental information in the report, or by creating environment.properties environment.xml file and store the file to the allure-results (this directory before generating the final html reports, generate dependent files directory) directory

environment.properties

Browser=Chrome
Browser.Version=63.0
Stand=Production
ApiUrl=127.0.0.1/login
python.Version=3.6

or

environment.xml

<environment>
    <parameter>
        <key>Browserkey>
        <value>Chromevalue>
    parameter>
    <parameter>
        <key>Browser.Versionkey>
        <value>63.0value>
    parameter>
    <parameter>
        <key>Standkey>
        <value>Productionvalue>
    parameter>
        <parameter>
        <key>ApiUrlkey>
        <value>127.0.0.1/loginvalue>
    parameter>
        <parameter>
        <key>python.Versionkey>
        <value>3.6value>
    parameter>
environment>

View report execution run.py

Categories

Test Report The default statistics are two types of test results, the failure of the use cases and test failures, we can add a custom statistical type cases, the same need to create categories.json file in the directory allure-results

[
  {
    "name": "Ignored tests",
    "matchedStatuses": ["skipped"]
  },
  {
    "name": "Infrastructure problems",
    "matchedStatuses": ["broken", "failed"],
    "messageRegex": ".*bye-bye.*"
  },
  {
    "name": "Outdated tests",
    "matchedStatuses": ["broken"],
    "traceRegex": ".*FileNotFoundException.*"
  },
  {
    "name": "Product defects",
    "matchedStatuses": ["failed"]
  },
  {
    "name": "Test defects",
    "matchedStatuses": ["broken"]
  }
]

View report execution run.py

Fixtures and Finalizers

Fixtures and Finalizers is pytest test method in the beginning and end of the test calls, allure call will automatically track each fixture, and a detailed display fixture and parameters which will call, and will retain the right to call Counts

Test code

test_allure_html.py

def function_scope_step():
    print("function_scope_step")


def class_scope_step():
    print("class_scope_step")


def module_scope_step():
    print("module_scope_step")


def session_scope_step():
    print("session_scope_step")


def step_inside_test_body():
    print("step_inside_test_body")


@pytest.fixture(params=[True, False], ids=['param_true', 'param_false'])
def function_scope_fixture_with_finalizer(request):
    if request.param:
        print('True')
    else:
        print('False')

    def function_scope_finalizer():
        function_scope_step()

    request.addfinalizer(function_scope_finalizer)


@pytest.fixture(scope='class')
def class_scope_fixture_with_finalizer(request):
    def class_finalizer_fixture():
        class_scope_step()

    request.addfinalizer(class_finalizer_fixture)


@pytest.fixture(scope='module')
def module_scope_fixture_with_finalizer(request):
    def module_finalizer_fixture():
        module_scope_step()

    request.addfinalizer(module_finalizer_fixture)


@pytest.fixture(scope='session')
def session_scope_fixture_with_finalizer(request):
    def session_finalizer_fixture():
        session_scope_step()

    request.addfinalizer(session_finalizer_fixture)


class TestClass(object):

    def test_with_scoped_finalizers(self,
                                    function_scope_fixture_with_finalizer,
                                    class_scope_fixture_with_finalizer,
                                    module_scope_fixture_with_finalizer,
                                    session_scope_fixture_with_finalizer):
        step_inside_test_body()

View report execution run.py

@allure.step

pytest support some modifications function test cases require the use of @ allure.step, the test cases to display more detailed testing process allure report

Test code

test_allure_feature.py modify the following code file

@allure.step("

Enter your user name

") def input_username(): print("

Enter your user name

") @allure.step("

enter password

") def input_password(): print("

enter password

")

View report execution run.py

conftest.py

@ Allure.step modified test procedures defined in conftest.py also supports file as step fixture, and now our new conftest.py file in the project directory, write the following code

conftest.py

@allure.step("

Open your browser

") def fixture_step(): pass @pytest.fixture def init_url(): fixture_step() yield True

test_allure_feature.py file, add the following use cases

    def test_init_url(self, init_url):
        flag = init_url
        assert flag == True

View report execution run.py

allure.attach

Use allure.attach can add files, pictures, log, html code, etc. to report. We modify test_allure_feature.py in the following use cases, and in cases where the directory with pictures to add attach.png

    def test_failed(self):
        """failed"""
        try:
            assert False
        except AssertionError as e:
            with open("attach.png", "rb") as f:
                context = f.read()
                allure.attach(context, "

Pictures error

", attachment_type=allure.attachment_type.PNG) raise e

View report execution run.py

@allure.description

If you want to show the test case descriptions in the report, then you can use @ allure.description (string) or @ allure.description_html (html code) decorate your test case, test_allure_feature.py file modify the following code

    @allure.description("

This is a test case has failed to perform

") def test_failed(self): """

You can also add a description of the use case here, but will be covered allure.description

""" try: assert False except AssertionError as e: with open("attach.png", "rb") as f: context = f.read() allure.attach(context, "

Pictures error

", attachment_type=allure.attachment_type.PNG) raise e

View report execution run.py

@allure.title

Use allure.title (title) in the name of the test case can rename allure report, test_allure_feature.py following code file modification

    @allure.title("

Login success scenario - {data}

") @pytest.mark.parametrize("data", login_success_data, ids=ids_login_success_data) def test_login_success(self, data): """

Login successful test

""" user = input_username(data["user"]) pwd = input_password(data["pwd"]) result = login(user, pwd) assert result == data["expected"]

@allure.link

@allure.testcase

@allure.issue

These three features can add a link to a test case, test_allure_feature.py file modify the following code

    @allure.testcase("https://www.cnblogs.com/linuxchao/", "

Test address

") def test_init_url(self, init_url): flag = init_url assert flag == True @allure.link("https://www.cnblogs.com/linuxchao/", name="

bug link

") @allure.description("

This is a test case has failed to perform

") def test_failed(self): """

You can also add a description of the use case here, but will be covered allure.description

""" try: assert False except AssertionError as e: with open("attach.png", "rb") as f: context = f.read() allure.attach(context, "

Pictures error

", attachment_type=allure.attachment_type.PNG) raise e @allure.issue("https://www.cnblogs.com/linuxchao/", "

Broken links

") def test_broken(self): """broken""" with open("broken.json", "r", encoding='utf8') as f: f.read()

View report execution run.py

 

 

 

@allure.feature

@allure.story

feature story and is known as behavior-driven mark, since the two markers, can be more clearly grasp the function of each test case and test cases for each test scenario Adoption of the report

Test class uses the @ allure.feature modified test method test_allure_feature.py file modifications using @ allure.story

View report execution run.py

Two or more markers can be displayed not only in the test report, but may also be performed using a command or the specified test scenario module

@allure.severity

This tag is used to identify test cases or test level class, divided blocker, critical, normal, minor, trivial5 levels, the following are the test cases by level mark, and look at the test report

to sum up

These are the most characteristic feature of all the allure-pytest plug-in support in pytest, perhaps finishing is not very detailed, so if you want a detailed understanding of the effect of specific features in the report, the need to try it yourself, attach This article reference links

https://docs.qameta.io/allure/

Categories
Uncategorized

Three.js development room (three)

The first three did not relate to the engine room, just a few fragmented knowledge, this section we began formal painting exterior walls.

First, I obviously understand what is the wall? In fact, that white is a rectangular, long uncertain, wide general is 40cm, which is two meters high, which is a simple wall, of course, a lot of the walls are windows, doors what, in fact, that is fixed in the cuboid location dig a hole, then put what we need parties, such as doors, windows.

We need a top view of a room for analysis before painting the wall, like on the picture below this room

(Picture from the network)

As shown in the picture, like, this room is very standard, is a very standard rectangular room, length 900cm, width 600cm, is the wall on the left side of the glass partition, there is a door,

Well, we can start to dry, we need to initialize a room Json structure and layout, pay attention to doors and windows can not overlap, there is a door where the door windows need to be divided into left and right door two arrays (of course, you can also write determining a plurality of operating, but more troublesome).

{
  houseWidth: 900,   // 房间长度
  houseHeight: 600,  // 房间宽
  angle: 45,         // 房间朝向
  wall: [
    {position:{x: 0, y: 0, endX: 900, endY: 0}, door: {isDoor: false}, windows: {isWindows:false}},
    {position:{x: 900, y: 0, endX: 900, endY: 600}, door: {isDoor: false},  windows: {isWindows: false}},
    {position:{x: 0, y: 600, endX: 900, endY: 600}, door: {isDoor: false}, windows: {isWindows:false}},
    {position:{x: 0, y: 0, endX: 0, endY: 600}, door: {isDoor: true, doorNum: 2, door_PointL [{x: 0, y: 200, endX: 0, endY: 400, doorDirection: 2}]},  windows: {isWindows: true, windows__Point: [{x: 0, y: 0, endX: 0, endY: 150}, {x: 0, y: 450, endX: 0, endY: 600}]}}
  ]
},

Then we started painting the floor, we will present to do the same floor and room size:

    createFloor() {
            let _self = this;
            this.imgRendering.load("地板的图片", texture => {
                texture.wrapS = texture.wrapT = THREE.RepeatWrapping;
                texture.repeat.set(8, 8);
                var floorGeometry = new THREE.BoxGeometry(this.houseWidth, this.houseHeight, 1);
                var floorMaterial = new THREE.MeshBasicMaterial({
                    map: texture,
                    side: THREE.DoubleSide
                });
                floorMaterial.opacity = 1;
                floorMaterial.transparent = true;
                var floor = new THREE.Mesh(floorGeometry, floorMaterial);
                floor.position.y = 0;
                floor.rotation.x = Math.PI / 2;

                _self.scene.add(floor);
            })
        }

FIG execution results are as follows:

Purple is the color I added to the entire Html, mainly easy viewing floor, then we started painting the wall, before we initialize a painted wall painted rectangular (window width and height are by default 1) function:

initLambert() {
  var cubeGeometry = new THREE.BoxGeometry(1, 1, 1);
  this.initLambertMod = new THREE.Mesh(cubeGeometry, this.wallMatArray);
};

After a good package when we would not have painted wall painting a wall to create a new geometry and material of each, we only need to clone we have just initiated the wall like

After we formally package having a specific length, angle and position on wall

/**
  * 画长方体
  * @param { 长方体的长度 } width 
  * @param { 长方体的高度 } height 
  * @param { 长方体的厚度 } depth 
  * @param { 长方体旋转的角度 } angle 
  * @param { 长方体的材质 } material 
  * @param { 长方体的X轴坐标 } x 
  * @param { 长方体的Y轴坐标 } y 
  * @param { 长方体的Z轴坐标 } z 
  */
createLambert(width, height, depth, angle, material, x, y, z) {
    var code = this.initLambertMod.clone();
    code.scale.set(width, height, depth)
    code.position.set(x, y, z);
    code.rotation.set(0, angle * Math.PI, 0); //-逆时针旋转,+顺时针
    return code;
};

In this way we will have a length and breadth, rectangular direction, position to draw out,

Just drawn is not enough, we need to link up data and models, we first this.data.wall specific information to traverse to get it the way it is, whether there are windows and doors, walls start and end points, and know the starting point end point, we can calculate how long this wall specifically, there is this wall angle

Above, we can have more than two points to derive information that line

Length: Math.sqrt (Math.pow (Math.abs (300 -0), 2) + Math.pow (Math.abs (0 -300), 2));

Angle: Math.asin ((300- 0) / (0 – 300)) / Math.PI

So that we know the specific information on that line, here we can draw a wall:

createHouseWall() {
  this.data.wall.map((item) => {
    var position = item.position;
    var w = position.endX - position.x;
    var h = position.endY - position.y;
    var x = (position.x + w / 2) - (this.houseWidth / 2);
    var z = (position.y + h / 2) - (this.houseHeight / 2);
    var width = Math.sqrt(Math.pow(w, 2) + Math.pow(h, 2));
    var angle = Math.asin(h / width) / Math.PI;
   
    if (item.windows.isWindows || item.door.isDoor) {
      // 有窗户或有门或都有
    } else {
      // 没门、没窗户
      let code = this.createLambert(width, 200, 10, angle, this.matArrayB, x, 100, z);
      this.scene.add(code);
    }
  });
};        

We can see the complete implementation of such results the following figure

Short of a wall, above both doors have windows, then we have to make both door windows, offer a picture look cool

To achieve this, we must first package a geometric ti cutting function:

/**
  * 几何体裁切函数
  * @param { 被采裁切的集合体 } bsp 
  * @param { 要裁掉的集合体 } less_bsp 
  * @param { 区分是机房的墙还是机柜裁切的 } mat 
  */
returnResultBsp(bsp, less_bsp, mat) {
        switch (mat) {
            case 1:
                var material = new THREE.MeshPhongMaterial({
                    color: 0x9cb2d1,
                    specular: 0x9cb2d1,
                    shininess: 30,
                    transparent: true,
                    opacity: 1
                });
                break;
            case 2:
                var material = new THREE.MeshPhongMaterial({
                    color: 0x42474c,
                    specular: 0xafc0ca,
                    shininess: 30,
                    transparent: true,
                    opacity: 1
                });
                break;
            default:
        }

        var sphere1BSP = new ThreeBSP(bsp);
        var cube2BSP = new ThreeBSP(less_bsp); //0x9cb2d1 淡紫,0xC3C3C3 白灰 , 0xafc0ca灰
        var resultBSP = sphere1BSP.subtract(cube2BSP);
        var result = resultBSP.toMesh(material);
        result.material.flatshading = THREE.FlatShading;
        result.geometry.computeFaceNormals(); //重新计算几何体侧面法向量
        result.geometry.computeVertexNormals();
        result.material.needsUpdate = true; //更新纹理
        result.geometry.buffersNeedUpdate = true;
        result.geometry.uvsNeedUpdate = true;
        if (mat == 2) {
            result.nature = "Cabinet";
        }
        return result;
    };

After we started to have a wall with a door or window begins processing, finishing first data, the data can be organized into what I can handle the simplest of

createHouseWall() {
  this.data.wall.map((item) => {
    var position = item.position;
    var w = position.endX - position.x;
    var h = position.endY - position.y;
    var x = (position.x + w / 2) - (this.houseWidth / 2);
    var z = (position.y + h / 2) - (this.houseHeight / 2);
    var width = Math.sqrt(Math.pow(w, 2) + Math.pow(h, 2));
    var angle = Math.asin(h / width) / Math.PI;
   
    if (item.windows.isWindows || item.door.isDoor) {
      // 有窗户或有门或都有
          // 当然判断里面还是分开成有门或者有窗户,但互不干涉
          var window__List = [];   // 盛放窗户的数组
          var door__List = [];       // 盛放门的数组
          if (item.windows.isWindows) {
            item.windows.windows__Point.map((windows__Point, window__index) => {
                 let window__Json = {};
                 let windows__w = windows__Point.endX - windows__Point.x;
let windows__h = windows__Point.endY - windows__Point.y; window__Json.window__x = (windows__Point.x + windows__w / 2) - (this.houseWidth / 2); window__Json.window__z = (windows__Point.y + windows__h / 2) - (this.houseHeight / 2); window__Json.window__width = Math.sqrt(Math.pow(windows__w, 2) + Math.pow(windows__h, 2)); window__Json.w_Height = 120; window__Json.window__y = 100; window__List.push(window__Json); }); }       if (item.door.isDoor) { var door__num = item.door.doorNum || 1; item.door.door_Point.map((door__Point, door__index) => { var door__Json = {}; var windows__w = door__Point.endX - door__Point.x; var windows__h = door__Point.endY - door__Point.y; if (door__num == 2) { let doubleDoorList = []; for (var i = 0; i < 2; i++) { door__Json = {}; door__Json.door__x = (door__Point.x + windows__w / 2) - (this.houseWidth / 2) + (door__Point.endX - door__Point.x) / 2 * i; door__Json.door__z = (door__Point.y + windows__h / 2) - (this.houseHeight / 2) + (door__Point.endY - door__Point.y) / 2 * i; door__Json.door__width = (Math.sqrt(Math.pow(windows__w, 2) + Math.pow(windows__h, 2))) / 2; door__Json.door__height = 180; door__Json.door__y = 100; door__Json.doorDirection = door__Point.doorDirection; if (door__Point.doorDirection < 2) { doubleDoorList.unshift(door__Json); } else { doubleDoorList.push(door__Json); } } door__List.push(doubleDoorList); } else { door__Json.door__x = (door__Point.x + windows__w / 2) - (this.houseWidth / 2); door__Json.door__z = (door__Point.y + windows__h / 2) - (this.houseHeight / 2); door__Json.door__width = Math.sqrt(Math.pow(windows__w, 2) + Math.pow(windows__h, 2)); door__Json.door__height = 180; door__Json.door__y = 100; door__Json.doorDirection = door__Point.doorDirection; door__List.push(door__Json); } }); }     } else {       // 没门、没窗户       let code = this.createLambert(width, 200, 10, angle, this.matArrayB, x, 100, z);       this.scene.add(code);     }   }); };

We will begin after finishing the above data to complete the operation, this time we need to create a function cerateWallHadDoorOrGlass to start drawing a glass door and a wall

//画有门和有窗子的墙(工具函数)
    cerateWallHadDoorOrGlass(width, height, depth, angle, material, x, y, z, door__list, windows__List) {
        //茶色:0x58ACFA   透明玻璃色:0XECF1F3
        var glass_material = new THREE.MeshBasicMaterial({
            color: 0XECF1F3
        });
        glass_material.opacity = 0.5;
        glass_material.transparent = true;
        var wall = this.returnLambertObject(width, height, depth, angle, material, x, y, z);
        windows__List.map((item, index) => {
            var window_cube = this.returnLambertObject(item.window__width, item.w_Height, depth, angle, material, item.window__x, item.window__y, item.window__z);
            wall = this.returnResultBsp(wall, window_cube, 1);
            let code = this.returnLambertObject(item.window__width, item.w_Height, 2, angle, glass_material, item.window__x, item.window__y, item.window__z);
            this.scene.add(code);
        });
        var status__result = [0.5, 0.5, 0, 0, ]
        door__list.map((item, index) => {
            if (item.length == 2) {
                item.map((c_item, c_index) => {
                    let door_cube = this.returnLambertObject(c_item.door__width, c_item.door__height, 10, angle, this.matArrayB, c_item.door__x, c_item.door__y, c_item.door__z);
                    wall = this.returnResultBsp(wall, door_cube, 1);
                    let doorgeometry = new THREE.BoxGeometry(100, 180, 2);
                    let door = "";
                    if (c_index == 0) {
                        door = new THREE.Mesh(doorgeometry, this.LeftDoorRenderingList);
                    } else {
                        door = new THREE.Mesh(doorgeometry, this.DoorRenderingList);
                    }
                    door.position.set(c_item.door__x, c_item.door__y, c_item.door__z);
                    door.rotation.y = status__result[c_item.doorDirection] * Math.PI;
                    door.nature = "door";
                    door.direction = c_item.doorDirection;
                    door.isClose = 1;
                    door.doorIndex = c_index;
                    this.scene.add(door);
                });
            } else {
                let door_cube = this.returnLambertObject(item.door__width, item.door__height, 10, angle, this.matArrayB, item.door__x, item.door__y, item.door__z);
                wall = this.returnResultBsp(wall, door_cube, 1);
                let doorgeometry = new THREE.BoxGeometry(100, 180, 2);
                let door = new THREE.Mesh(doorgeometry, this.DoorRenderingList);
                door.position.set(item.door__x, item.door__y, item.door__z);
                door.rotation.y = status__result[item.doorDirection] * Math.PI;
                door.nature = "door";
                door.direction = item.doorDirection;
                door.isClose = 1;
                this.scene.add(door);
            }

        });
        this.scene.add(wall);
    };

So, you're done, we put a side door but did not have glass walls look

Painted wall on this here, in this article I spent a whole afternoon, the project is directly from vue init webpack dome began, spectators if you feel okay, trouble to a "recommended", ha ha ha, when the whole I did an afternoon of hard work in vain! * _ *

 

Categories
Uncategorized

8. Function – closure

First, the function name

⼀ function name is variable, but it is ⼀ a special variable, and can be performed with parentheses variable ⾏ function.

1. The memory address of the function name

def func():
    print("呵呵")
    print(func)
结果:

2. The function name can be assigned to other variables

def func():
    print("呵呵")
    print(func)
# 把函数当成⼀个变量赋值给另⼀个变量
a = func
# 函数调⽤ func()
a() 

3. The function name can be used as function parameters

def func():
    print("吃了么")
def func2(fn):
    print("我是func2")
    fn() 
    print("我是func2")
# 把函数func当成参数传递给func2的参数fn.
func2(func) 

4. The name can be used as the return value of the function

def func_1():
    print("这⾥是函数1")
def func_2():
    print("这⾥是函数2")
    print("这⾥是函数1")
    return func_2
# 执⾏函数1. 函数1返回的是函数2, 这时fn指向的就是上⾯函数2
fn = func_1()
# 执⾏上⾯返回的函数
fn()    

Second, closure

1. Definitions

If an internal function, the variable to be applied externally to (but not in the global scope) references, then the internal function is considered to be closure.

Closure (the Closure) lexical closures (Lexical Closure) for short, is a function of the free variables cited. This free variables referenced and this function will exist together, even if it has left the creation of an environment is no exception. Therefore, the closure is a combination of entities associated with a function and a reference environment formed.

Popular talk is: If an internal function, the pair (but not in the global scope) were variable reference outside the scope, then the internal function is considered to be closure (closure). It is just a “inner” function by a name (variable) to refer to, and the name (variable) for “outer layer” contains in terms of its function, it is a local variable.

The main purpose of closure: realized decorator

2. Closure use

    Closure function must have built-in functions

    Namespace references an inline function requires the nested function variables

    Closures function must return the built-in functions

(1) Normal closure

def func(a, b):
    def line(x):
        return a * x - b
    return line

line = func(2, 3)
print(line(5))

(2) modification value closure

def func(a, b):
    def line(x):
        nonlocal a
        a = 3
        return a * x - b
    return line

line = func(2, 3)
print(line(5))
Categories
Uncategorized

[Original] (c) Linux paging_init resolve

background

    Read the fucking source code! –By Lu Xun

    A picture is worth a thousand words. –By Golgi

Description:

    Kernel Version: 4.14

    ARM64 processor, Contex-A53, binuclear

    Use tools: Source Insight 3.5, Visio

1 Introduction

From (b) Linux physical memory initialization, call paging_init seen before, stored Kernel Image and DTB of two physical memory area can be visited (the corresponding page table has been created). Although the physical memory memblock_add been added to the system, but this part of the physical memory to virtual memory mapping has not been established, it can memblock_alloc allocate some physical memory, but can not access, all need to wait for the implementation of paging_init. After the final page table is set up, you can go to access the final physical address of the virtual address.

按照惯例,先上图,来一张ARM64内核的内存布局图片吧,最终的布局如下所示:

Adventure open it!

2. paging_init

paging_init dapper source code directly Tieshanglai sub module to introduce it.

/*
 * paging_init() sets up the page tables, initialises the zone memory
 * maps and sets up the zero page.
 */
void __init paging_init(void)
{
    phys_addr_t pgd_phys = early_pgtable_alloc();   /********(mark 1)*******/
    pgd_t *pgd = pgd_set_fixmap(pgd_phys);

    map_kernel(pgd);                                        /********(mark 2)*******/
    map_mem(pgd);                                         /********(mark 3)*******/

    /*
     * We want to reuse the original swapper_pg_dir so we don't have to
     * communicate the new address to non-coherent secondaries in
     * secondary_entry, and so cpu_switch_mm can generate the address with
     * adrp+add rather than a load from some global variable.
     *
     * To do this we need to go via a temporary pgd.
     */
    cpu_replace_ttbr1(__va(pgd_phys));                 /********(mark 4)*******/
    memcpy(swapper_pg_dir, pgd, PGD_SIZE);
    cpu_replace_ttbr1(lm_alias(swapper_pg_dir));

    pgd_clear_fixmap();
    memblock_free(pgd_phys, PAGE_SIZE);

    /*
     * We only reuse the PGD from the swapper_pg_dir, not the pud + pmd
     * allocated with it.
     */
    memblock_free(__pa_symbol(swapper_pg_dir) + PAGE_SIZE,
              SWAPPER_DIR_SIZE - PAGE_SIZE);
}

    mark 1: the size of physical memory allocated to store a PGD;

    mark 2: the respective core segments are mapped;

    mark 3: Add the physical memory subsystem memblock mapping;

    mark 4: switching the page table, the page table of contents and replace swappper_pg_dir newly created table of contents page;

代码看起来费劲?图来了:

It will be further analysis of each sub-module below.

3. early_pgtable_alloc

这个模块与FIX MAP映射区域相关,建议先阅读前文(二)Linux物理内存初始化
先上图:

FIX MAP area division may be seen from FIG.
    This function will be assigned physical memory, then before borrowing global page table bm_pte, to establish a physical address to map the virtual address, the role of this map is order to access physical memory, the memory is cleared, so it’s just a temporary operation, after the operation, calls pte_clear_fixmap () to clear the map.

After early_pgtable_alloc, we see paging_init call pgd_set_fixmap function, the function call after by memblock_alloc allocate physical memory, will eventually be used to store pgd table, the contents of this area will be copied to the last swapper_pg_dir go.

4. map_kernel

map_kernel的主要工作是完成内核中各个段的映射,此外还包括了FIXADDR_START虚拟地址的映射,如下图:

映射完成之后,可以看一下具体各个段的区域,以我自己使用的平台为例:

The address information can also be found from the System.map file.

aarch64-linux-gnu-objdump -x vmlinux can view more detailed address information.

5. map_mem

从函数名字中可以看出,map_mem主要完成的是物理内存的映射,这部分的物理内存是通过memblock_add添加到系统中的,当对应的memblock设置了MEMBLOCK_NOMAP的标志时,则不对其进行地址映射。
map_mem函数中,会遍历memblock中的各个块,然后调用__map_memblock来完成实际的映射操作。先来一张效果图:

map_mem are mapping a physical address to a linear region, we also found the Kernel Image in text, rodata segment mapped twice, because other subsystems, such as hibernate, mapped to the linear region, may require linear region the address referenced text kernel, rodata, the time limit will become read-only mapping / unenforceable, prevent accidental modification or execution.

map_kernelmap_mem函数中的页表映射,最终都是调用__create_pgd_mapping函数实现的:

Overall, it is progressively page table mapping relationship, while the middle will control privileges.
    Details will not be repeated, a code reading with pictures, the effect will be better Oh.

6. Replace the memory page table and release

This small part of the code, not on the map, look at the code it:

    /*
     * We want to reuse the original swapper_pg_dir so we don't have to
     * communicate the new address to non-coherent secondaries in
     * secondary_entry, and so cpu_switch_mm can generate the address with
     * adrp+add rather than a load from some global variable.
     *
     * To do this we need to go via a temporary pgd.
     */
    cpu_replace_ttbr1(__va(pgd_phys));
    memcpy(swapper_pg_dir, pgd, PGD_SIZE);
    cpu_replace_ttbr1(lm_alias(swapper_pg_dir));

    pgd_clear_fixmap();
    memblock_free(pgd_phys, PAGE_SIZE);

    /*
     * We only reuse the PGD from the swapper_pg_dir, not the pud + pmd
     * allocated with it.
     */
    memblock_free(__pa_symbol(swapper_pg_dir) + PAGE_SIZE,
              SWAPPER_DIR_SIZE - PAGE_SIZE);

In simple terms, the new-established pgd page table of contents, copied to swapper_pg_dir, that is before the overwrite temporary page tables. When the copy is complete, it is clear, we can paging_init the beginning of physical memory allocated to freed.
    In addition, in the previous article also analyzed over time swapper_pg_dir page table stored, is stored in a row pgd, pud, pmd, and now just need to reuse swapper_pg_dir, the rest of the course can be released.

Well, go beyond that, the road is long, from the Buddy System, Slab, Malloc and a variety of memory seems to show the operation is still a long way, he continued.

Categories
Uncategorized

selenium reptiles

Web automated testing tools can be run in a browser, the browser operation according to the instruction, is only a tool to be used in conjunction with third-party browser, compared to before the study reptiles just a little bit slower. And what this method crawling do not care when ajax dynamic loading mechanism and other anti-climb. So look for the label directly F12 can find, without determining the source code exists.

installation

Linux: sudo pip3 install selenium

Windows: python -m pip install selenium

phantomjs browser

phantomjs browser interface, also known as non-browser (also known as the headless browser), a page is loaded in memory to run efficiently.

Installation (phantomjs (no browser interface), chromedriver (Google Chrome), geckodriver (Firefox))

Windows

1, download the corresponding version of phantomjs, chromedriver, geckodriver

2, chromedriver download the Google browser corresponding version of the chromedriver.exe copied to the Scripts directory python installation directory (added to the system environment variable) to view the python installation path: where python

3, verify, cmd command line: chromedriver

Linux

1, must extract: tar -zxvf geckodriver.tar.gz

2, copy the file to extract the / usr / bin / (add environmental variables): sudo cp geckodriver / usr / bin /

3, change permissions

  sudo -i

  cd /usr/bin/

  chmod 777 geckodriver

Sample code: using selenium + Google browser to open Baidu, and Baidu Home screenshots

from selenium import webdriver

browser = webdriver.Chrome()            #

Create a browser object

browser.get('http://www.baidu.com/') #

Open Baidu

browser.save_screenshot('baidu.png') #

Screenshots

browser.quit() #

Exit browser

Sample Code 2: Open Baidu, search Zhao Liying

from selenium import webdriver
import time

#

Create a browser object - it has opened the browser

browser = webdriver.Chrome() browser.get('http://www.baidu.com/') #

Open Baidu

ele = browser.find_element_by_xpath('//*[@id="kw"]') #

Find the search box

ele.send_keys('

Zhao Liying

') #

Send a text to the search box: Zhao Liying

time.sleep(1) #

Baidu to find a button, click

browser.find_element_by_xpath('//*[@id="su"]').click() time.sleep(2) browser.quit() #

Close the browser

 

browser browser object method

    browser = webdriver.Chrome (executable_path = ‘path’) path drives the address to the browser

    browser.get (url) open path Path

    browser.page_source: view the content of the response (page source code)

    browser.page_source.find ( ‘string’): Searching html specified string from source code is not found Return: -1

    browser.quit (): Close your browser

Find elements

Find a single element (a node object)

  1. browser.find_element_by_id(”)
  2. browser.find_element_by_name(”)
  3. browser.find_element_by_class_name(”)
  4. browser.find_element_by_xpath(”)
  5. browser.find_element_by_link_text(”)
  6. … …

Multi-element lookup ([node object list])

  1. browser.find_elements_by_id(”)
  2. browser.find_elements_by_name(”)
  3. browser.find_elements_by_class_name(”)
  4. browser.find_elements_by_xpath(”)
  5. … …

Node object manipulation

    .send_keys ( ”) transmits the content search box

    .click () Click

    .text get the text content

    .get_attribute ( ‘src’) retrieving properties

    .find ( “”) to find the string in the response

from selenium import webdriver

browser = webdriver.Chrome()
browser.get('https://www.qiushibaike.com/text/')

#

Find a single element

div = browser.find_element_by_class_name('content') print(div.text) #

Multi-element look for: [, ]

divs = browser.find_elements_by_class_name('content') for div in divs: print('*************************') print(div.text) print('*************************') browser.quit() #

Exit browser

Jingdong reptiles Case

Destination URL: https: //www.jd.com/ crawl target: product name, product price, quantity evaluation, commodity merchants

Ideas remind

    Jingdong open to the Product Search page

    Matching a list of all goods node objects

    The text node objects taken out to see the law, is there a better approach?

    After a complete extraction, if the judgment is not the last one, then click Next

Implementation steps

Find node

    Home Search box: // * [@ id = “key”]

    Home Search Button: // * [@ id = “search”] / div / div [2] / button

    Object list node commodity product information page: // * [@ id = “J_goodsList”] / ul / li

JS script execution, access to dynamic load data

  browser.execute_script(‘window.scrollTo(0,document.body.scrollHeight)’)

from selenium import webdriver
import time


class JdSpider(object):
    def __init__(self):
        self.i = 0
        self.url = 'https://www.jd.com/'
        self.browser = webdriver.Chrome()

    #

Get information page - the page specific commodities

def get_html(self): self.browser.get(self.url) self.browser.find_element_by_xpath('//*[@id="key"]').send_keys('

Reptile book

') #

Search box, enter "Reptile book"

self.browser.find_element_by_xpath('//*[@id="search"]/div/div[2]/button').click() #

Click Search

time.sleep(3) #

To the commodity page load time

#

Parsing the page

def parse_html(self): #

Pull the pull-down menu at the bottom, JS script execution

self.browser.execute_script('window.scrollTo(0,document.body.scrollHeight)') time.sleep(2) #

Extract a list of all goods node objects li list

li_list = self.browser.find_elements_by_xpath('//*[@id="J_goodsList"]/ul/li') for li in li_list: info_list = li.text.split('\n') if info_list[0].startswith('

Each full

') or info_list[1].startswith(''): price = info_list[1] name = info_list[2] comment = info_list[3] shop = info_list[4] elif info_list[0].startswith('

Single

'): price = info_list[3] name = info_list[4] comment = info_list[5] shop = info_list[6] else: price = info_list[0] name = info_list[1] comment = info_list[2] shop = info_list[3] print(price, comment, shop, name) #

The main function

def main(self): self.get_html() while True: self.parse_html() #

Determine whether the click Next, did not find the explanation is not the last page

if self.browser.page_source.find('pn-next disabled') == -1: self.browser.find_element_by_class_name('pn-next').click() time.sleep(2) else: break print(self.i) if __name__ == '__main__': spider = JdSpider() spider.main()

provided no interface mode chromedriver

from selenium import webdriver

options = webdriver.ChromeOptions()   #

No interface settings

options.add_argument('--headless') #

Add no interface parameters

browser = webdriver.Chrome(options=options) browser.get('http://www.baidu.com/') browser.save_screenshot('baidu.png') browser.quit()

The above code to interface mode without

from selenium import webdriver
import time


class JdSpider(object):
    def __init__(self):
        self.url = 'https://www.jd.com/'
        self.options = webdriver.ChromeOptions()  #

No interface settings

self.options.add_argument('--headless') #

Add no interface parameters

#

It can create a normal browser object

self.browser = webdriver.Chrome(options=self.options) self.i = 0 #

Commodity statistics

#

Get information page - the page specific commodities

def get_html(self): self.browser.get(self.url) self.browser.find_element_by_xpath('//*[@id="key"]').send_keys('

Reptile book

') #

Search box, enter "Reptile book"

self.browser.find_element_by_xpath('//*[@id="search"]/div/div[2]/button').click() #

Click Search

time.sleep(3) #

To the commodity page load time

def parse_html(self): #

The progress bar is pulled to the bottom, so that all data is dynamically loaded

self.browser.execute_script('window.scrollTo(0,document.body.scrollHeight)') time.sleep(2) #

Wait for the dynamic data load is complete

#

Extract a list of all goods node objects li list

li_list = self.browser.find_elements_by_xpath('//*[@id="J_goodsList"]/ul/li') item = {} for li in li_list: #

find_element: Find a single element

item['name'] = li.find_element_by_xpath('.//div[@class="p-name"]/a/em').text.strip() item['price'] = li.find_element_by_xpath('.//div[@class="p-price"]').text.strip() item['comment'] = li.find_element_by_xpath('.//div[@class="p-commit"]/strong').text.strip() item['shop'] = li.find_element_by_xpath('.//div[@class="p-shopnum"]').text.strip() print(item) self.i += 1 def main(self): self.get_html() while True: self.parse_html() #

Determine whether the last page

if self.browser.page_source.find('pn-next disabled') == -1: self.browser.find_element_by_class_name('pn-next').click() time.sleep(3) else: break print('

Number of Products:

', self.i) self.browser.quit() if __name__ == '__main__': spider = JdSpider() spider.main()

View Code

Keyboard

from selenium.webdriver.common.keys import Keys
​
browser = webdriver.Chrome()
browser.get('http://www.baidu.com/')
#

1, in the search box, enter "selenium"

browser.find_element_by_id('kw').send_keys('

Zhao Liying

') #

2, enter a space

browser.find_element_by_id('kw').send_keys(Keys.SPACE) #

3, Ctrl + a simulation Select

browser.find_element_by_id('kw').send_keys(Keys.CONTROL, 'a') #

4, Ctrl + c analog copy

browser.find_element_by_id('kw').send_keys(Keys.CONTROL, 'c') #

5, Ctrl + v paste Analog

browser.find_element_by_id('kw').send_keys(Keys.CONTROL, 'v') #

6, press enter, instead of the Search button

browser.find_element_by_id('kw').send_keys(Keys.ENTER)

Mouse operation

import time
from selenium import webdriver
#

Importing mouse events

from selenium.webdriver import ActionChains browser = webdriver.Chrome() browser.get('http://www.baidu.com/') #

Find the "Settings" node

element = browser.find_element_by_xpath('//*[@id="u1"]/a[8]') #

Move the mouse to set node, move_to_element ()

actions = ActionChains(browser) actions.move_to_element(element) actions.perform() #

perform () is actually performing operations

time.sleep(1) #

Find Advanced Settings node, and click

browser.find_element_by_link_text('

Advanced Search

').click()

Switch pages

Application and page links midpoint open a new page of the website appears, but the object browser or the browser before the page object

all_handles = browser.window_handles get all current handle (window)

browser.switch_to_window (all_handles [1]) to switch to a new browser window, a new window object acquires

Ministry of Civil Affairs website

The civil division code crawled into the database, according to the hierarchy (sub-table – the provincial table, table city, county table)

Build a database table

# 建库
create database govdb charset utf8;
use govdb;
# 建表
create table province(
        p_name varchar(20),
        p_code varchar(20)
        )charset=utf8;
        create table city(
        c_name varchar(20),
        c_code varchar(20),
        c_father_code varchar(20)
        )charset=utf8;
        create table county(
        x_name varchar(20),
        x_code varchar(20),
        x_father_code varchar(20)
        )charset=utf8;

Thinking

    selenium + Chrome open a page, two pages and extract new links

    Incremental crawling: version to compare the database tables, and, before determining whether the climb (if an update)

    If there is no update, prompt the user directly without continued crawling

    If the data in the table is updated, deleted before re-crawling and inserted into the database table

    After final completion: disconnect from the database, close the browser

from selenium import webdriver
import pymysql


class GovSpider(object):
    def __init__(self):
        #

No interface settings

options = webdriver.ChromeOptions() options.add_argument('--headless') self.browser = webdriver.Chrome(options=options) #

Add parameters

self.one_url = 'http://www.mca.gov.cn/article/sj/xzqh/2019/' #

Create a database and related variables

self.db = pymysql.connect('localhost', 'root', '123456', 'govdb', charset='utf8') self.cursor = self.db.cursor() #

Create a list of three for executemany () to insert records in table 3

self.province_list = [] self.city_list = [] self.county_list = [] #

Get home, and extract the secondary pages link (false links)

def get_incr_url(self): self.browser.get(self.one_url) #

Extracts the latest link, to determine whether an incremental crawl

td = self.browser.find_element_by_xpath('

// td [@ class = "arlisttd"] / a [contains (@title, "Code")]

') #

Extract links and databases do comparison, determine whether you need both how the crawl

#

get_attribute () will auto-complete extraction link

two_url = td.get_attribute('href') #

result is affected by the number of pieces returned

result = self.cursor.execute('select url from version where url=%s', [two_url]) if result: print('

Without crawling

') else: td.click() #

Switch handle

all_handlers = self.browser.window_handles self.browser.switch_to.window(all_handlers[1]) self.get_data() #

Data Capture

#

The URL address is stored in version table

self.cursor.execute('delete from version') self.cursor.execute('insert into version values(%s)', [two_url]) self.db.commit() #

Two pages to extract administrative division code

def get_data(self): #

Benchmark xpath

tr_list = self.browser.find_elements_by_xpath('//tr[@height="19"]') for tr in tr_list: code = tr.find_element_by_xpath('./td[2]').text.strip() name = tr.find_element_by_xpath('./td[3]').text.strip() print(name, code) #

Analyzing hierarchy, added to the corresponding database table (correspondence table field)

# province: p_name p_code # city : c_name c_code c_father_code # county : x_name x_code x_father_code #

To add data to the corresponding table

if code[-4:] == '0000': self.province_list.append([name, code]) if name in ['

Beijing

', '

Tianjin

', '

Shanghai

', '

Chongqing

']: self.city_list.append([name, code, code]) elif code[-2:] == '00': self.city_list.append([name, code, (code[:2] + '0000')]) else: if code[:2] in ['11', '12', '31', '50']: self.county_list.append([name, code, (code[:2] + '0000')]) else: self.county_list.append([name, code, (code[:4] + '00')]) #

# For loop with indented, all data after climbing unified excutemany (),

#

Perform database insert statement

self.insert_mysql() def insert_mysql(self): #

Be sure to delete table records 1. Update

self.cursor.execute('delete from province') self.cursor.execute('delete from city') self.cursor.execute('delete from county') #

2. Insert the new data

self.cursor.executemany('insert into province values(%s,%s)', self.province_list) self.cursor.executemany('insert into city values(%s,%s,%s)', self.city_list) self.cursor.executemany('insert into county values(%s,%s,%s)', self.county_list) #

3. Submit to the database to perform

self.db.commit() print('

Data Capture completed successfully stored in the database

') def main(self): self.get_incr_url() self.cursor.close() #

Once all the data processed is disconnected

self.db.close() self.browser.quit() #

Close the browser

if __name__ == '__main__': spider = GovSpider() spider.main()

SQL command exercises

1. All provinces and counties query (multi-table query implementation)

select province.p_name,city.c_name,county.x_name from province,city,county  where province.p_code=city.c_father_code and city.c_code=county.x_father_code;

2. All provincial cities and counties query (join query realization)

select province.p_name,city.c_name,county.x_name from province inner join city on province.p_code=city.c_father_code inner join county on city.c_code=county.x_father_code;

Web client authentication

You can fill in the URL address

url = ‘http: // username: password @ address normal’

Example: crawling a day notes

from selenium import webdriver
​
url = 'http://tarenacode:[email protected]/AIDCode/aid1904/15-spider/spider_day06_note.zip'
browser = webdriver.Chrome()
browser.get(url)

iframe sub-frame

iframe sub-frame for the web page in nested, this should iframe to switch to the sub-frame, then perform other operations.

browser.switch_to.iframe(iframe_element)

Example – Login qq-mail

import time
from selenium import webdriver

browser = webdriver.Chrome()
browser.get('https://mail.qq.com/cgi-bin/loginpage')

#

Find iframe sub-frame and switch to this iframe

login_frame = browser.find_element_by_id('login_frame') browser.switch_to.frame(login_frame) #

qq + Password + Log

browser.find_element_by_id('u').send_keys('

account number

') browser.find_element_by_id('p').send_keys('

password

') browser.find_element_by_id('login_button').click() time.sleep(5) #

Reservation page record time

#

Extract data

ele = browser.find_element_by_id('useralias') print(ele.text)

 

Categories
Uncategorized

08 UA application pool and the agent pool in Scrapy

Download Middleware Introduction

In Scrapy, there is a component between the engine and the downloader called Downloader Middlewares. Since it is a hook somewhere between Scrapy’s request/response processing, there are 2 things to do:

(1) passes the request to the engine during downloading, download middleware Requests for a series of processes. Such as setting the User-Agent request, setting agent like ip

(2) When the downloader completes passing Response to the engine, download middleware can do a series of processing on Responses. such as gzip decompression, etc.

 

Download middleware has the following functions are managed

– process_request request when the method is called by downloading the middLEware

– Process_response results are processed by this method when downloaded through middlware

– Process_exception is called when an exception occurs during download

 

When writing middlers, you need to think about which method to write the function to be implemented most suitable for processing in which process. Middlware can be used to process requests, process results or combine the use of some methods of signal coordination, etc., can also be added to the original crawler to adapt to the project, which can also be written in the extension, in fact the extension is more decoupled, it is recommended to use the extension. In crawlers, mainly use download middlware to process requests, usually set a random User-Agent for the request, and set a random proxy ip. An anti-crawler strategy designed to prevent crawling websites.

 

I. UA pool: User-Agent pool

– role: as many requests scrapy project disguised as a different type of browser identity.

– Operating flow:

1. interception request download middleware

2. Tamper and disguise UA in the request header information intercepted to the request

3. Turn on the download middgets in the configuration file

 

Middleware.py part code shows:

from scrapy.contrib.downloadermiddleware.useragent import UserAgentMiddleware        #

Guide package

import random #

UA write code pool (UA pool to separate the package into a class)

class RandomUserAgent(UserAgentMiddleware): def process_request(self, request, spider): ua = random.choice(user_agent_list) request.headers.setdefault('User-Agent',ua) #

The current intercepted write operation request ua

user_agent_list = [ "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3", "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 " ]

 

II. Agent pools

– role: as many as possible will scrapy project requests to a different IP settings.

– Operating procedures:

1. interception request download middleware

2. Modify the IP of the intercepted request to a proxy IP

3. Open the downloaded middleware in the configuration file

 

Middleware code display: bulk ip replacement of intercepted requests, separate package download middleware class

class Proxy(object):

def process_request(self, request, spider):

        #

Judging by the URL intercepted to the request (whether the protocol header is http or https), request.url return value: http://www.xxx.com

h = request.url.split(':')[0] #

The protocol header of the request

if h == 'https': ip = random.choice(PROXY_https) request.meta['proxy'] = 'https://'+ip else: ip = random.choice(PROXY_http) request.meta['proxy'] = 'http://' + ip #

Proxy IP can be selected

PROXY_http = [ '153.180.102.104:80', '195.208.131.189:56055', ] PROXY_https = [ '120.83.49.90:9000', '95.189.112.214:35508', ]

Proxy ip is usually done when the request is unsuccessful, so we can write proxy ip to process_exception later.

 

Three, UA agent pool in pools and exemplary middleware

To the wheat fields of real estate, for example, will display the code below and shown in detail how to use the pool and UA agent pool in Scrapy framework.

   

 

Reptile File: maitian.py

import scrapy
from houseinfo.items import HouseinfoItem               #

Import item

class MaitianSpider(scrapy.Spider): name = 'maitian' # start_urls = ['http://bj.maitian.cn/zfall/PG{}'.format(page for page in range(1,101))] start_urls = ['http://bj.maitian.cn/zfall/PG100'] #

Parsing function

def parse(self, response): li_list = response.xpath('//div[@class="list_wrap"]/ul/li') for li in li_list: item = HouseinfoItem( title = li.xpath('./div[2]/h1/a/text()').extract_first().strip(), price = li.xpath('./div[2]/div/ol/strong/span/text()').extract_first().strip(), square = li.xpath('./div[2]/p[1]/span[1]/text()').extract_first().replace('',''), area = li.xpath('./div[2]/p[2]/span/text()[2]').extract_first().strip().split('\xa0')[0], adress = li.xpath('./div[2]/p[2]/span/text()[2]').extract_first().strip().split('\xa0')[2] ) yield item #

to the pipeline, which then defines the storage method

items file: items.py

import scrapy

class HouseinfoItem(scrapy.Item):
    title = scrapy.Field()          #

Store headers, which can store any type of data

price = scrapy.Field() square = scrapy.Field() area = scrapy.Field() adress = scrapy.Field()

Pipelines.py file: pipelines.py

class HouseinfoPipeline(object):
    def __init__(self):
        self.file = None

    #

When starting to crawl, perform once

def open_spider(self,spider): self.file = open('maitian.csv','a',encoding='utf-8') #

Selected append mode

self.file.write(",".join(["

Title

","

Monthly rent

","

area

","

region

","

Address

","\n"])) print("

Begin to crawl.

") #

Because this method is called multiple times, the opening and closing operations of the file are written in the other two methods that will only be executed once.

def process_item(self, item, spider): content = [item["title"], item["price"], item["square"], item["area"], item["adress"], "\n"] self.file.write(",".join(content)) return item #

At the end of the crawler, perform once

def close_spider(self,spider): self.file.close() print("

End the crawler.

")

Middleware file Middlewares.py

from scrapy import signals

class HouseinfoDownloaderMiddleware(object):

    #

UA pool

user_agent_list = [ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 " "(KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1", "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 " "(KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 " "(KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6", "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 " "(KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 " "(KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 " "(KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5", "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 " "(KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3", "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3", "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3", "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3", "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 " "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 " "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24" ] PROXY_http = [ '153.180.102.104:80', '195.208.131.189:56055', ] PROXY_https = [ '120.83.49.90:9000', '95.189.112.214:35508', ] def process_request(self, request, spider): #

Set up a requested UA using a UA pool

request.headers['User-Agent'] = random.choice(self.user_agent_list) return None def process_response(self, request, response, spider): return response #

Intercept the request object exception occurred

def process_exception(self, request, exception, spider): if request.url.split(':')[0] == 'http': request.meta['proxy'] = 'http://'+random.choice(self.PROXY_http) else: request.meta['proxy'] = 'https://' + random.choice(self.PROXY_https)

Profile: settings.py

# -*- coding: utf-8 -*-
BOT_NAME = 'houseinfo'

SPIDER_MODULES = ['houseinfo.spiders']
NEWSPIDER_MODULE = 'houseinfo.spiders'

# Obey robots.txt rules ROBOTSTXT_OBEY = False #

Open pipeline

ITEM_PIPELINES = { 'houseinfo.pipelines.HouseinfoPipeline': 300, #

The value of 300 is the priority, the smaller the value, the higher the priority

}

# Open the downloaded middleware

DOWNLOADER_MIDDLEWARES = { 'houseinfo.middlewares.HouseinfoDownloaderMiddleware': 543, }

 

Categories
Uncategorized

[JavaScript] All-you-can-eat series of JavaScript simulates multithreading concurrency

Preamble

Recently, learning is a fiery topic, and I, but also want to be a kind of scholar, that is, to put JavaScript and multithreaded concurrency these two eight sticks, to hard together, also wrote a concurrent-thread-js library. Awkward, when I found out the irrational thing in it, that is, what exactly is the application scenario for this stuff, I found that I’ve finished writing the code.

 
 
 

⚠️ Note! In this article refers to the thread function is asynchronous with JS simulated “false thread,” not in the sense of a true multi-threading, please do not misunderstand ⚠️

 
 

github address

https://github.com/penghuwan/concurrent-thread.js

The purpose of this paper

In fact, the library is of little use, but in the process of writing, I have developed a new understanding of the use of Promise, Async functions and event flow, and I am also learning and understanding how to write a non-business, generic npm module from scratch, so I want to share it with you. The real purpose of the text.

 

Okay, let’s start with a story.

scene one

Scene II

 

github address

https://github.com/penghuwan/concurrent-thread.js​github.com

Attention! Without considering webworker such a solution, we generally assume that JS is single-threaded.

concurrent-thread-js Features

Implement concurrency coordination functions, semantic, naming and role for single-threaded JavaScript referencing Java implementation, mention APIs such as sleep/join/interupt and lock and conditional variables, and provide functionality for inter-thread communication, rely on ES6 syntax, based on Promise and Async function implementation, therefore require Babel compilation to run. JavaScrPt is inherently single-threaded, so this is only a simulation at the API level. In the introduction below, each so-called thread is actually a normal asynchronous function, and on this basis implements the coordination of different threads.

 

Why not choose webworker achieve?

Yes, in general JS mock multithreading we may choose webworker, but it must require you to manually create additional webworker script files and use them through new work (‘work.js’), which does not achieve the API I want in my project, and note: the environment in webwork is not window! A lot of methods you can’t call. You can only take this kind of scheme, i.e. doing the function in the main thread, which is another reason I didn’t choose webworker.

Say this to say, but in fact most of the time or use webworker enough

 

When to use concurrent-thread-js

The problem is really the soul of torture, but since the code is written is written, and how I have to make up a reason to come out ah! amount. . . Let me see Ha

What it does is: you can use this concurrency mock library when JS projects need to let two functions not interfere with each other’s execution, and don’t want them to block the main thread, at the same time, but also want the two functions to achieve a similar requirement of coordination between concurrent multithreads…. Does this Nima have this application scenario?! (KIND).

 

API Overview

  • submit (function, [namespace]): receives a function, normal function or Async function, and executes “thread” asynchronously

  • sleep (ms): “Thread” sleep, sleep time can be specified MS, in milliseconds

  • join (ThreadName): “Thread” synchronization, the “Thread” function that calls this method will continue to execute after the ThreadName execution ends

  • interrupt (thread Name): “thread” interrupted affect “thread” internal call this.isInterrupted () return value

  • Lock.lock: lock, a time only one “thread” function to enter the critical section, other “threads” function to wait, fair and non-locking, that lined up behind the thread function does not have, in a random manner competition.

  • Lock.unlock: lift the unfair lock

  • Condition.wait: No execution condition, “thread” enters the waiting state and waits to be waken

  • Condition.notify: random wake of a wait “threads”

  • Condition.notifyAll: not written, wake up all the wait “threads”

  • getState: not finished acquiring “Thread” state, including RUNNALE (run), WAITING (waiting), BLOCKED (blocking), TERMINATED (termination)

Three classes: ThreadPool, Lock and Condition

Our API is written into three classes, respectively

  • ThreadPool class: Contains SubMb/Sleep/Join/Interrupt/getState method

  • Lock Class: Contains Lock.lock and Lock.unlock methods

  • Condition class: and Condition.notify method comprising Condition.wait

Note: The following “thread” refers to the asynchronous function that is simulated in JS

A1.submit method

submit an analog submitted to the thread pool thread

//

Note: For a step-by-step introduction, the following is a simplified code

//

The state of each thread function of storage, such as whether to interrupt and thread status

const threadMap = {}; class ThreadPool { //

Simulating Thread Interruption

interrupt(threadName) { } //

Analog Thread Synchronization

join(threadName, targetThread) { } //

Analog thread sleep

sleep(ms) { } }; function submit(func, name) { if (!func instanceof Function) return; //

Mode 1: pass in a named function; Mode 2: pass in the second parameter, the thread namespace

const threadName = func.name || name; //

ThreadMap is responsible for storing thread state data

threadMap[threadName] = { state: RUNNABLE, isInterrupted: false }; //

Let func asynchronous calls, while incoming function prototype scope bindings for ThreadPool

Promise.resolve({ then: func.bind(ThreadPool.prototype); }) }

 

First of all, we have done three things:

  1. Get the namespace of the thread function, and initialize the thread initial data state stored by the different threads global threadMap

  2. Take the submitted function func as the then argument of a thenable object in the Promise.Resolve method, which is equivalent to immediately “complete” a Promise, while executing func in the then method, func will execute asynchronously rather than synchronously, you can simply understand that it is similar to the execution of SetTimeout (func,0);

  3. The func scope bindings for ThreadPool instance of the new generation, ThreadPool defined above us to introduce our methods, such as sleep / join / interupt, etc., what good is it? This means that we can directly in the function by way of calling this.interrupt to call our API definition, and in line with our habits (note, in addition to ordinary function outside the class defined function arrow actually stored in the prototype )

submit(async function example() {
    this.interrupt();
});

 

But the problem is: now because all functions are called by this method in the ThreadPool prototype, we need to pass in the “thread” identity in the asynchronous function, such as the thread name. This is obviously inconvenient and not elegant, for example the following thread function named example

submit(async function example() {
    this.interrupt('example');
});

 

With this module users will be surprised: I obviously function in the example, why the name parameter example to call the method passed? ? Can not this thing inside the module did it?

Correct! Here is what we do this thing, we write a delegateThreadPool method, by its function name for the ThreadPool agency to handle different “threads” function

//

Returns ThreadPool after Agent

function delegateThreadPool(threadName) { //

threadName is pending thread name, incoming call when the submit method

//

ThreadPool after the proxy

const proxyClass = {}; //

Get all the original methods of ThreadPool, assign the props array

var props = Object.getOwnPropertyNames(ThreadPool.prototype); for (let prop of props) { //

Acting ThreadPool, threadName increase this parameter for all methods

let fnName = prop; proxyClass[fnName] = (...args) => { const fn = baseClass[fnName]; return fn(threadName, ...args); }; } return proxyClass; } function submit(func, name) { //

omit other code...

const proxyScope = delegateThreadPool(threadName); //

Let func asynchronous call, do not block the main thread, while achieving concurrency

Promise.resolve({ then: function () { //

Func for the ThreadPool to bind this subject after the proxy to invoke methods

func.call(proxyScope); } }); } //

When you call this.sleep method, has been named as a parameter function without increasing the

submit(async function example() { this.interrupt(); });

 

In other words, our thread function func is not bound ThreadPool.prototype, but rather the object returned after delegateThreadPool treatment: proxyScope. At this time, we in the “thread” in the body of the function is invoked this.interrupt method, has been named as a parameter without increasing function, because this work, proxyScope objects help us to do, in fact, it worked quite simple – it’s every functions, in a return to the closure of the same name which calls ThreadPool function, passing the thread name as the first parameter.

A2.Sleep method

Role: Thread hibernation

sleep is very simple, nothing more than a return Promise instance, in tune setTimeOut Promise function inside, so time to resolve the function execution, during which time the modified Promise await sentence will block for some time, resolve and then await the statement went down execution, and we want to meet the effects of sleep

//

Analog "thread" sleep

sleep(ms) { return new Promise(function (resolve) { setTimeout(resolve, ms); }) } //

Submit “Thread”

submit(async function example() { //

Obstruction stay three seconds before output 1

await this.sleep(3000); console.log(1); });

 

A3. interrupt method

Role: thread interrupts can be used to stop processing thread and other operations

Let’s introduce the interrupt method in Java: In Java, you can’t stop a thread by calling terminate method, because this may cause data inconsistency problems due to sudden interruptions in the processing logic, so use the interrupt method to position an interrupt flag as true, and then pass the The iInterrupted method breaks out of key code as a judgment condition.

So for mock, I do the same for dealing with “thread” breaks in JS, but the root reason we do this is: we don’t have a way to stop a thread function at all! (JAVA has but is not allowed to use, that is, it is only deprecated)

    //

Simulating Thread Interruption

interrupt(threadName) { if (!threadName) { throw new Error('Miss function parameters') } if (threadMap[threadName]) { threadMap[threadName].isInterrupted = true; } } //

Get thread interrupt status

isInterrupted(threadName) { if (!threadName) { throw new Error('Miss function parameters') } //

!!! is: turn undefined to false

return !!threadMap[threadName].isInterrupted; }

 

A4. join method

join (ThreadName): “Thread” synchronization, the “Thread” function that calls this method will continue to execute after the ThreadName execution ends

The join method is the same as the sleep method above. We let it return a promise. As long as we don’t change resolve, then the await statement that is externally decorating the promise will be suspended until the other thread of join is executed. Let’s see the timing! Give the promise to resolve, at which point the external modification of the await statement can be executed down again?

 
 

But the question is: How do we implement “feature complete implementation of a function to another function notice it,” this? Yes! That is our JavaScript favorite routines: the event flow! Here we use the generic event-emitter module to achieve this end before and after the event flow.

We only triggered when any function ends the end of the event (join-finished), while passing the function name of the thread as a parameter, and then listen for the event in the internal join method, and response time to call resolve method is not to be proud.

 

The first is listening for the end event of the thread function inside the join method

import ee from 'event-emitter';
const emitter = ee();
//

Analog Thread Synchronization

join(threadName, targetThread) { return new Promise((resolve) => { //

Listening for end events for other thread functions

emitter.on('join-finished', (finishThread) => { //

According to the end of the thread thread name to make a judgment finishThread

if (finishThread === targetThread) { resolve(); } }) }) }

 

Also trigger join-finished event at the end of thread function execution, passing thread name as argument

import ee from 'event-emitter';
const emitter = ee();
function submit(func, name) {
   // ...
    Promise.resolve({
        then: func().then(() => {
          emitter.emit('join-finished', threadName);
        })
    });
}

Use as follows:

submit(async function thread1 () {
  this.join('thread2');
  console.log(1);
});
submit(async function thread2 () {
  this.sleep(3000);
  console.log(2)
})
//

After 3s, output 2 1 in turn

A5. Lock.lock & Lock.unlock (unfair lock)

We mainly want to write two methods: lock and unlock methods. We need to set a Boolean property isLock

  • lock method: lock method will first determine whether isLock is false, and if so, on behalf of occupation no thread critical section, the thread is allowed to enter the critical section, while the isLock set to true, does not allow other threads to enter the function. When other threads to enter, because the judge isLock true, the setTimeOut recursive call from time to time determine whether isLock is false, so consumed by way of a lower performance simulation while an endless loop. When they detect isLock is false when it will enter the critical section, while isLock set to true. Because the thread behind in no particular order, so this is a fair and non-lock

  • Unlock method: Unlock is set the isLock property to false. Unlock it.

//

This is an unfair lock

class Lock { constructor() { this.isLock = false; } //

Lock

lock() { if (this.isLock) { const self = this; //

Loop while loop, constantly testing isLock is equal to false

return new Promise((resolve) => { (function recursion() { if (!self.isLock) { //

Holding the lock

self.isLock = true; //

External await the statement continues down

resolve(); return; } setTimeout(recursion, 100); })(); }); } else { this.isLock = true; return Promise.resolve(); } } //

Unlock

unLock() { this.isLock = false; } } const lockObj = new Lock(); export default lockObj;

 

The run example is as follows:

async function commonCode() {
    await Lock.lock();
    await Executor.sleep(3000);
    Lock.unLock();
}

submit(async function example1() {
    console.log('example1 start')
    await commonCode();
    console.log('example1 end')
});
submit(async function example2() {
    console.log('example2 start')
    await commonCode();
    console.log('example2 end')
});

 

Export

//

Output immediately

example1 start example2 start //

Output after 3 seconds

example1 end //

Three more seconds to output

example2 end

 

A6. Condition.wait & Condition.Notify (conditional variable)

  • Condition.wait: does have the condition, the thread enters the waiting state, waiting to be awakened

  • Condition.Notify: Wake Thread

Sorry! To write here, I really have a dry mouth, I can not write anymore, but the truth is the same as before:

Nothing more than: Event listening+Promise+Async function combo punch, set to get

import ee from 'event-emitter';
const ev = ee();

class Condition {
    constructor() {
        this.n = 0;
        this.list = [];
    }
    //

When the conditions are not satisfied, so that the thread is in a wait state

wait() { return new Promise((resolve) => { const eventName = `notify-${this.n}`; this.n++; const list = this.list; list.push(eventName); ev.on(eventName, () => { //

Remove an Event Name from a List

const i = list.indexOf(eventName); list.splice(i, 1); //

Getting the external function back to execution

debugger; resolve(); }) }) } //

Select a thread to wake up

notify() { const list = this.list; let i = Math.random() * (this.list.length - 1); i = Math.floor(i); ev.emit(list[i]) } }

 

Test code

async function testCode() {
    console.log('i will be wait');
    if (true) {
        await Condition.wait();
    };
    console.log('i was notified ');
}

submit(async function example() {
    testCode();
    setTimeout(() => {
        Condition.notify();
    }, 3000);
});

Export

i will be wait
//

Output after 3 seconds

i was notified

 

The last large sum

Actually, in the end, what I want to share with you is not a concurrency ah, what a lot of threads.

In fact, I want to say is: an event listener + Promise + Async function well with this combination of boxing ah

  • You want a piece of code stop by? OK! Promise to write a return of function, modified by await, it stopped it!

  • You want to control it (await) do not stop, and continue to go down? OK! Promise to resolve the fall, it’s gone down

  • You say you do not know how to control it to stop, because the code transmitter and monitor the distribution of events in two places? OK! Then use the event stream

 

This article finished, the following is the entire project code (just write an article only to find there are bug, will be changed to change)

Categories
Uncategorized

NOTE Using java summary

In 2005, sun launched a jdk1.5, while the launch of the annotation features to attract the attention of many people, using annotations to code, java programmers can alleviate the suffering of the tedious configuration.

Use annotations can write easier to maintain, bug less code.

What comment is it? According to the official statement, notes that meta tags can be added to your code, and applied to package declarations, type declarations, constructors, methods, fields, parameters and variables.

Notes provides a very useful way to show whether the method you write that rely on other methods, whether they are complete, whether to write the class refer to other classes, and so on.

According to Oracle’s official statement, based on notes written in the java code template code is generated based on the source code comments, so we avoid writing boilerplate code in most cases. This leads to a declarative programming style, in this style, the programmer says what to do, the utility will write the appropriate code to implement it.

Briefly, the annotation is a mechanism for meta tags associated with the program element, and allows the virtual machine compiler or extracted from the annotation program behavior elements, and generate the code interdependent when necessary.

Now for our java annotations learning journey.

Built-in notes
    java will be built some of which have achieved good notes, you can directly use the built-in annotation is mainly used to provide java compiler directive.

There are five built-in java annotations:

  • @Deprecated
  • @Override
  • @SuppressWarnings
  • @SafeVarargs
  • @FunctionalInterface

@Deprecated mainly used to annotate classes, methods, variables marked with labels is not recommended to use, if you use the code does not recommend the use of classes, methods, variables, the compiler will give you a warning.
    The following are examples of the use @Deprecated:

public class AnnotationTest {
    public static void main(String args[]){
        MyAnnotation myAnnotation = new MyAnnotation();
        int age = myAnnotation.age;
        myAnnotation.eat();
    }
}

@Deprecated 
class MyAnnotation {
    @Deprecated
    int age;
    
    @Deprecated
    void eat(){
        
    }
}

Defines a MyAnnotation, are used on the class name, variable and method names @Deprecated comment.

When using the identification @Deprecated classes, methods, or variables,

@Deprecated there is another use, javadoc is stated in the document class, method or variable Why is not recommended, and alternative methods are given:

@Deprecated
/**
  @deprecated 已废弃,请使用MyNewComponent.
*/
class MyComponent {

}

@Override annotations on the method used, the identity of the parent class override this method. If this method is inconsistent, and the parent class method, the compiler will display an error.

It is strongly recommended to use the method in the parent class @Override rewriting notes, do not have any effect, but if you do not use @Override comment when someone changes the method of the parent class, you will not be able to identify whether the sub-category method rewrite the parent class. The use of @Override notes, as long as the parent does not have this method, the compiler will prompt an error does not correspond to the parent class method.
    Here is an example using @Override annotation:

class Anaimal{
    public void run(){
        
    }
}

class Cat extends Anaimal{
    @Override
    public void run(){
        
    }
}

RUN Cat class () method uses @Override annotations, if the method to runl Cat class (), the compiler is displayed The method run1 of type Cat must override or implement a supertype error ().

If no @Override annotation, a method to runl Cat class (), the system is not being given.

When @SuppressWarnings comment also on the method for suppressing warnings of deprecated method call or by type of conversion unsafe, the compiler will issue some warnings, use @SuppressWarnings you can ignore those warnings.
    Example of use:

@SuppressWarnings
public void methodWithWarning() {


}

@SafeVarargs annotation is mainly used for suppression parameter type safety warning, this is jdk1.7 new features.
    Example of use:

@SafeVarargs
static void testSafeVarargs(List ... stringLists) {
    Object[] array = stringLists;
    List tmpList = Arrays.asList(42);
    array[0] = tmpList; 
    String s = stringLists[0].get(0);
}

If you do not use @SafeVarargs annotation, the compiler will give a warning: Type safety: Potential heap pollution via varargs parameter stringLists.
    Use @SafeVarargs there is a premise, you must ensure that the use of a variable length parameter method, the type of security problem does not occur when used with generic classes. Otherwise running row will throw a ClassCastException.

@SafeVarargs annotations only on the method satisfies the following conditions:

    Variable-length method or constructor parameter.

    The method must be declared as static or final.

@FunctionalInterface annotation is mainly used to compile the level of error checking, with the comment, when you write the interface does not meet the functional interface definition, the compiler will complain.
    Example of use:

@FunctionalInterface
interface GreetingService
{
    void readMessage(String message);
}

Creating notes
    Annotations and create an interface somewhat similar, are using the word customs seized interface, except that when you create a comment, you need to add a @ character in front of the interface.

Here is an example to create a comment:

public @interface MyAnnotation {
}

Creating top notes do not have any member variables.

Create notes with a member variable:

@interface TestAnnotation{
    int age();
    String name();
}

TestAnnotation annotation has two member variables, age and name.
    Annotated member variable declarations in a non-parametric method no method body form.

Use the default keyword specifies the default value annotation member variable:

@interface TestAnnotation{
    int age() default  2;
    String name() default  “小明”;
}

When using custom annotations, annotation if the variable has a default value, you can not specify a value for the member variable, direct use of default values.
    Example of use:

@interface TestAnnotation{
    int age() default 2;
    String name() default "小明";
}


class MyAnnotation {
    @TestAnnotation
    public void getInfo(){
        
    }
}

If the variable comment no default, you must specify values ​​for each variable in the use of all, the code is as follows:

@interface TestAnnotation{
    int age();
    String name();
}


class MyAnnotation {
    @TestAnnotation(age=2,name="小明")
    public void getInfo(){
        
    }
}

Yuan notes
    What is a meta-annotation it? Annotation is the annotation metadata annotations, annotation is used to define the annotation, the annotation can be understood as basic data types. This stuff is really hard to pronounce, or look at the code more comfortable.

@Target(ElementType.METHOD)
@interface Test_Target {
   public String doTestTarget();
}

@Target meta-annotation is used to define Test_Target comment.

java provides five yuan notes, are:

    @Retention specify annotations life cycle, namely survival time.

    Documents @Documented javadoc command generated reflect the content of the comment

    @Target specified annotation element which may be used, such as classes, methods, variables,

    @Inherited inheritance annotations,

    Notes @Repeatable reusable

@Retention annotation is used to specify the life cycle, namely survival time. @Retention provides the following three values, @Retention in use, which must be of a value.

    RetentionPolicy.SOURCE comment only remain in the source stage, discarded when compiling the generated class file compiler, annotation information can not be obtained through reflection.

    RetentionPolicy.CLASS notes are only kept to be compiled when it will not be loaded into the JVM, annotation information can not be obtained by reflection, which is the default.

    RetentionPolicy.RUNTIME annotation can be retained when the program is running, it will be loaded into the JVM, you can get to them through reflection program is running.

Here is an example of a custom annotations @Retention definitions:

@Retention(RetentionPolicy.RUNTIME)
@interface Test_Retention{
    
}

@Test_Retention
class MyAnnotation {
    public void getInfo(){
        
    }
}

The above code creates a Test_Retention annotation, and the annotation can be used to retain @Retention to specify Test_Retention program runs (RetentionPolicy.RUNTIME), MyAnnotation use @Test_Retention modified so that at runtime annotation information can be acquired by reflecting MyAnnotation.

@Documented If the class A notes annotation using @Documented yuan notes, then use javadoc generated classes A document will contain a corresponding annotation information.
    Example of use:

@Documented
@interface TestDocument{
    String doTestDocument();
}

@TestDocument (doTestDocument="保留注解信息测试")
class MyAnnotation {
    
    public void getInfo(){
        
    }
}

@Target the modified annotation specifies the range of the object, it can be used for arguments, parameters, methods, and other package information.
    @Target meta-annotation provides the following eight values:

    It is used to describe the type of annotation ElementType.ANNOTATION_TYPE

    The method of construction for the annotation ElementType.CONSTRUCTOR

    ElementType.FIELD for variable comment

    ElementType.LOCAL_VARIABLE for local variables notes

    ElementType.METHOD method for annotation

    ElementType.PACKAGE package for comment

    ElementType.PARAMETER process parameters for the annotation

    ElementType.TYPE for classes, interfaces, enumerations comment

Code Example:

@Target(ElementType.METHOD)
@interface TestMethodTarget{
    
}

@Target(ElementType.FIELD)
@interface TestFieldTarget{
    
}

@Target(ElementType.TYPE)
@interface TestTypeTarget{
    
}

TestMethodTarget only annotation method for annotation class, TestFieldTarget only for annotation class member variables, TestTypeTarget can be used for annotation class, interface (including type of annotation), or an enum declaration

@Inherited specify annotations can be inherited. @Inherited a class using the modified notes, then the notes will be used subclass of the class.
    Code Example:

@Inherited
@interface TestInherited{
    
}

@Repeatable same annotation can be used multiple times. For example, a person has a variety of hobbies, running, painting, watching movies and so on.
    Sample code:

@interface Persons {
    Person[]  value();
}


@Repeatable(Persons.class)
@interface Person{
    String hobby default "";
}


@Person(hobby="runing")
@Person(hobby="drawing")
@Person(hobby="watching movies")
public class Tom{

}

The above code, @ Repeatable annotated Person. @Repeatable brackets behind the class corresponding to one container annotation.
    What is the container notes it? The place is used to store other annotations. It is itself a comment.

Details of the above knowledge, and custom annotation syntax annotations. Now we look at how to use annotations for unit testing.

Use these notes to be reflected knowledge, knowledge of another article on reflection I look at “Java reflection using the summary.” After getting reflected by the object, the object method to invoke isAnnotationPresent contains the specified types of notes. Annotation

Sample code:

public class Marathonrunner {
    @RuningTest
    public void test5km(){
        System.out.println("进行5公里跑步测试");
    }

    public void test10km(){
        System.out.println("进行10公里跑步测试");
    }

    @RuningTest
    public void test21km(){
        System.out.println("进行21公里跑步测试");
    }

    public void test42km(){
        System.out.println("进行42公里跑步测试");
    }

}

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
@interface  RuningTest{

}


public class RuntestTool {
    public static void main(String args[]){
        Marathonrunner xiaoming = new Marathonrunner();

        Class clazz = xiaoming.getClass();

        Method[] methods = clazz.getDeclaredMethods();
        for (Method method : methods) {
            if(method.isAnnotationPresent(RuningTest.class)){
                try {
                    method.setAccessible(true);
                    method.invoke(xiaoming, null);

                } catch (Exception e) {

                }
            }
        }
    }
}

operation result:

进行21公里跑步测试
进行5公里跑步测试

The above code, we define a RuningTest notes, which do not have any variable. This annotation using @Retention and @Target meta-annotation modification, where the value of @Target meta-annotation of the provisions of this RuningTest comment only on the method, and @Retention meta-annotation value specifies the information at run time can get to annotations.

Defines a class mobilization Marathonrunner marathon far, there are four methods.

It defines a class designed for testing Marathonrunner athletes approach. If we want to test a class, simply add @RuningTest comment on that class, without @RuningTest annotated method will not be tested. test5km () and test21km () methods are added @RuningTest notes, they were tested.

Notes in spring, mybatis annotation GF application. Next spring application specifically written articles in speaking notes under.

to sum up:
    This article explains the concept notes, the concept of yuan notes, and how to customize annotations, notes and how to use their own definition of unit testing.

Categories
Uncategorized

Cloud server anti-hacking attack and defense Record (II)

4, Quest trace hackers

4.1 fable apartment building

Corner of the city there is a single apartment building, the apartment door with a window. At night, the owner returned from work key in the door into the room, closed rest lights, usually half open windows, ventilation, keep the air fresh.

Like a cloud server for a single apartment. Hackers like as uninvited guests, uninvited, into the house hackers have two paths, one way is to open the door strutted in, the other way is Fanchuang into the room, how many need some Feiyanzoubi effort.

Apartment doors more advanced, using electronic locks, you need to enter the correct password to open the door into the house. Hackers do not know the password, you can not come in.

Single apartment in a 16th floor room 1605, Xiao Ming lived in this small apartment a year, and recently a friend SGX Xiaofang, this time stay in the apartment. There are already late from work on Xiao Ming and new friends, so Bob gave her a new account, set up a new password. Xiao Ming a little lazy, the account number and password is set to the same, and very few people thought the 16th floor, the corridor has a 24-hour monitoring, understanding him no one would dare trespassing.

After a month, two people arguing, Xiaofang moved out. Xiao Ming in a bad mood, alone, depressed, withdrawn forget to Xiaofang account password.

Nearby communities to a foreign tramp. Tramp is a highly intelligent person, like unrestrained and live undisciplined lives. One day, wandering tramp in the apartment building, walked into the room 1605, saw a girl head silhouette on the door, but also on the pattern of letters “fang”, when it is left loving Xiao Ming and Xiao Fang Xiu under. Wanderers had an idea, not many people take the name of friends and relatives to do your password? And try locks. This test does not matter, actually opened the apartment door.

So, this tramp was admitted to a single apartment, and Bob had played a wrong peak style “cohabitation” life. Small clear day to go to work, go home at night to live. Tramp over live in an apartment during the day, myself cook, eat and sleep, to leave before dark. Later, the tramp is not met, the evening also came to the apartment, the owner Xiao Ming fear of discovery, secretly Xiaoming drink the cup put a sleeping pill, dissolves without a trace, Xiao Ming drinking boiled water under the sleeping pills at night I will sleep very fragrant, very heavy. Then, in the evening it is dominated by the tramp.

So “cohabitation” a month later, Xiao Ming found some anomalies in the fridge a few less inexplicable eggs, Kleenex on the coffee table is also used very quickly, taking the garbage in the morning, come back at night the trash and eating out the rest of the chicken bones.

Xiao Ming put these anomalies small clearing with his good friend said, a small clearing is a well-known private detective, cracked many difficult cases. Xiao Ming received the news, immediately came to a small clearing Xiaoming apartment, inside and out investigation again, and then Bob went to work. Small clear eyes remain on the apartment door girl silhouette and “fang”, seemed to understand something. Enter the password, the apartment door opened automatically. Xiaoqing went to the video control room of the building, access to the surveillance video outside Xiaoming apartment and saw that were coming and going every day tramp. A small clearing modify the apartment door password, and called the police. After the police investigation, it issued an arrest warrant.

Night, Xiao Ming back, a small clearing to lay it all happened told Xiao Ming, Xiao Ming was shocked, then laughed, very pleased to have such a small clearing detective friend.

4.2 Query visitors Chi

As mentioned earlier, a cloud server is like a single apartment, and hackers like to break into the apartment tramp. Can enter from the door into the apartment, but also Fanchuang come into the house from the windows of high-rise apartments, a large number of difficult, but also pull the steel mesh windows.

From the door into the house, like on a cloud server remotely over SSH logging in, the owner is a cloud server administrator, also came through SSH login. External exposure to a cloud server service port, it can respond to an interview request return, but would not be happy and welcome intruder sneaked into the port and take over from the server. Fanchuang into the house, like attacking the cloud server service port, vulnerable ports to get in, it is very difficult Fanchuang

For the stranger, the door from the house is not easy, unless you know the password or access control access control destroyed.

There is no stranger broke into the cloud server, log on to double the system log to clear. Single apartment as access to video surveillance, who visited a glance.

The last command of Linux queries the most recent SSH login log on the cloud server.

# last
 

Figure System login log for Linux server

 

4.3. Find the difference.

A few years ago, we played a little game called Finding Differences: two pictures are full of objects or geometries of different shapes, two pictures are mostly the same, only a few different shapes and hidden in the two pictures. Looking for traces from the logbook is a bit like playing a different game. Quickly find different results, find differences, have new discoveries.

The source address for most logins is 192.168.x.x, is the cloud data center LAN address, and the login user is root, consistent with the administrator’s daily management habits.

There was a strange IP address record that caught my attention, the login account was jira, and the last exit was April 8 04:09 to 04:19, exactly as the last modification time of Trojan horse sd-pam “Apr 8 04:19”.

It can be inferred that hackers use jira user is logged into the system, implanted Trojans in 10 minutes, and then sped away, quietly waiting for “chicken” host returns data mining (digital currency).

 

4.4 discovered no secret code

Hackers is how to know the user name and password?

We know, SSH encrypted communication application layer protocols, all communication is through high-strength encryption information, it is difficult to decipher, the possibility to intercept the ciphertext then deciphered almost zero. The possibility of brute force password guessing is also zero. The possibility of brute force password guessing is also zero.

Hackers are not the administrator login account jira common root account, the account password is not further evidence comes to decipher.

There is another possibility, the administrator password is lost because of carelessness, using a simple password is easy to guess, such as: 123,123456, abc, or account passwords altogether equal.

Let’s verify, I guess weak passwords is established. So far, I do not know the account password jira.

Log in to JIRA cloud server through a springboard machine root. Then, switch to another regular user jira1 (the account login is limited, not remote login), root user to switch to ordinary users do not have to enter a password. Next, switching from the ordinary user to user jira1 jira, jira user must enter a password. Enter the password jira, switching actually succeeded to prove that the account password is jira jira! ! ! Here I express silent.

# ssh [email protected]192.168.1.x
# su – jira1
$ su - jira

Cloud server has a pair of vent secret, account number and password just like plain text, like a password leak than the apartment, leaving others free access to come and go. Apartment compromised, into a tramp; server security defense was broken, “mining” the Trojans are implanted, round the clock to dig a digital virtual currency.

 

4.5 locked doors compromised

Query system is not within the normal process of the user in use jira, jira user to confirm temporarily idle useless. Immediately amend jira user password and locked account.

# passwd jira
# usermod -L jira

And toured the operating system again, / etc / passwd, / etc / shadow have no landed users. SSH scored from the inlet port of the cloud server have been blocked.

After doing these things, after the preliminary judging system has been safe, my heart gradually ease down.

Made immediately after work on Monday with the cloud server administrator, system management team made the cloud server cluster vulnerability scanning and security reinforcement.

 

4.6 explore trace hackers

The mood relaxed, thinking only idle hacker came from, how come, but also do something!

Baidu query, hacking the source address of April 8 from Norway. However, the hacker does not have to hide in Norway, even a little relationship with Norway at all. Since the invasion of source address may be just a springboard for manipulation by a “chicken” host only.

 

Since the May 1, another wave of hackers struck in broad daylight (14:34 pm to 14:41) blatantly intrusive cloud server to be six minutes, but did not modify the wave of hackers left on the Trojans program. May the source address of the hacker from Switzerland, who would he be? Is the wave of hackers, visited the old battlefield came back to win. Or new to the “ethical” hackers, to see people gain an advantage, quietly leave. Unknown, but anything is possible.

 

Looking further ahead, January 10 to 18, there have been four login records, address sources vary, 185.112.82.230,185.112.82.235,41.231.5.34 and 188.166.171.252, respectively, from Russia, Russia, Tunisia and Netherlands.

Diversification of source address, seems to indicate that hackers can freely access this cloud servers. What is the reason to attract hackers flock to it? Why is jira user, not other users?

The answer is a port scan and public information learned through the scan, and then attack the server vulnerabilities.

 

4.7 Interpretation of hacker surgery

Hacker scan public IP address or domain name of the scan HTTP port (port 80, 8080), analysis of the HTML text returned, after filtration, extraction candidate words, and then compared with the thesaurus keywords. If the keyword match, extracts the corresponding application from keywords associated libraries, concluded that the address of the HTTP port provides the application services. For example: jira extracted keywords, to infer the JIRA server provides services; sonarqube to extract keywords, conclude that the server provides SonarQube services, and so on.

 

The default page Figure HTTP services exposed the specific services

 

Figure default HTTP service response returned exposed the specific services

 

The general public application installation guide, most users will be prompted to create a new account with the same name of the operating system, such as jira, mysql and redis etc., in order to install a common user applications. Some careless administrator, the system will give an account set up simple password. This is what this type of hack attacks one of the key reasons can succeed.

IP addresses on the Internet vast, why is this cloud server?

Stupid approach is based on IP addresses one by one scan and found an open port on the IP address, and then attack, break after break port, obtain system privileges, implanted Trojans.

A country allocated IP address segment number is fixed, unless there is a new IP address number section. National IP authorities to give their domestic cloud service provider assigned IP address numbers section, which is fixed, even publicly available any of a number of IP address segment of cloud service providers. With a cloud security infrastructure cloud providers are shared, with the same shield, also have the same flaw, if they find a loophole on a cloud server, it is possible that other cloud servers have similar vulnerabilities.

Rent virtual servers to the cloud service provider content provider, due to the safety management system, basic security software and security management team in common, its management of cloud server cluster will have a similar safety and similar vulnerabilities. Replicability, Zhaomaohuahu attacker captured a cloud server, it is easy to build on the attack more cloud servers.

Speaking earlier, the hacker implanted Trojan program is “mining” program. In the next episode will be the “mining” procedures surgery, a few chunks of dismembered program, analyze its internal operation mechanism, external communications contact mode and so on.

 

This article is an original work of authorship, originally published in a public anti-hacker invasion offensive and defensive Record No. cloud server (b)