astinchoi

[astinchoi]

Twisted] 10. Thread and Subprocess

Twisted doesn't make the code automatically non-asynchronous or non-blocking code. Twisted provides common networking, file system, timer activity non-blocking primitives and wrapping non-blocking api provided by OS.

Twisted is event-driven based. That means using callback and different architecture with synchronous program. It provides abstraction called 'deffered' helping manage callback.

However, sometimes it needs to use thread or process. 1) Thread There are some cases which cannot use callback and deffered. For example, it’s using 3rd party API which calls blocking during executing. So use it in thread. - callInThread: Execute in separated private thread (Preferred for using United Interface) - deferToThread: Same and the result returns to Deferred.

https://github.com/AstinCHOI/book_twisted/blob/master/ch10_thread_and_subprocess/blocking.py

in order to wait, not to shutdown for reactor before deferToThread finished, write reactor.stop of callback chain. https://github.com/AstinCHOI/book_twisted/blob/master/ch10_thread_and_subprocess/blocking_revised.py

- Others callFromThread callMultipleInThread blockingCallFromThread 2) Subprocess Twisted provides independent API to execute subprocess with non-blocking through reactor. 2-1) Execute a subprocess and get a result Example) remote man page https://github.com/AstinCHOI/book_twisted/blob/master/ch10_thread_and_subprocess/manpage_server.py # Terminal 1 $ python manage_server.py

# Terminal 2 $ telnet localhost 8000 > man ls - getProcessOutput: spawn subprocess and return a firing deffered when process finished. getProcessOutput(executable, args=(), env={}, path=None, reactor=None, errortoo=False) - getProcessValue(executable, args=(), env={}, path=None, reactor=None)

2-2) Custom process protocol See IProcessProtocol > ProcessProtocol - spawnProcess method (similar to getProcessOutput, but has richer syntax)

Example) Execute an echo application using EchoProcessProtocol and kill server after 10 sec. https://github.com/AstinCHOI/book_twisted/blob/master/ch10_thread_and_subprocess/twisted_spawnecho.py

# Terminal 1 $ python twisted_spawnprocess.py # Terminal 2 $ telnet localhost 8000

- ConnectionMade: Process starts as soon as it is called and transport is set to communicate this process self.transport.write method: write to stdin self.transport.writeToChild: appoint file descriptor. - outReceived: calls when data arrived at stdout and connected pipe. data is being buffering and arrived by chunk. so it’s appropriate to stack data until processEnded would be called. errReceived method is similar to this without stderr - inConnectionLost, outConnectionLost, errConnectionLost When stdin, stdout, stderr is closed, these methods are called. - processExited and processEnded: When all file descriptors are closed, processExited is callback called. ProcessEnded is appropriate place to end reactor. In Example: SIGTERM signal : graceful end KILL signal: - 3) Others http://twistedmatrix.com/documents/current/core/howto/threading.html http://twistedmatrix.com/documents/current/core/howto/gendefer.html http://twistedmatrix.com/documents/current/core/howto/process.html http://twistedmatrix.com/documents/current/core/examples/ https://launchpad.net/ampoule

[astinchoi]

Twisted] 9. Autenthication

Twisted Cred Terms. 1) Credentials The information to identify and verify user (usually username/password) >> twisted.cred.credentials.ICredentials

2) Avatar is server application business logic object. It's behaviour and data that user can do. ex) Mail server: mailbox / Web server: web resource / SSH server: remote shell >> zope.interface.Interface

3) Credentials checker Object to verify Credential through File, database or memory >> twisted.cred.checker.ICredentialsChecker >> twisted.cred.checkers.ANONYMOUS

4) Realm Object which provides the way to any possible avatar within application. 5) Portal is like window between User and Cred. Example 1) using variable/file https://github.com/AstinCHOI/book_twisted/blob/master/ch9_authentication/echo_cred.py 2) using DB https://github.com/AstinCHOI/book_twisted/blob/master/ch9_authentication/echo_server.py https://github.com/AstinCHOI/book_twisted/blob/master/ch9_authentication/echo_server.py

3) twistd application authentication ref. Chapter 6.  https://github.com/AstinCHOI/book_twisted/tree/master/ch9_authentication/echoproject and just changed Plug-in. https://github.com/AstinCHOI/book_twisted/tree/master/ch9_authentication/echoproject/twisted/plugins

(in this case, check with /etc/passwd and /etc/shadow)

Link: Twisted's Web in 60 seconds https://twistedmatrix.com/documents/current/web/howto/web-in-60/

[astinchoi]

Twisted] 8. Database

Do not make blocking call in main-thread, because in event loop it runs twisted application. Otherwise, entire event loop will stop.

twisted provides API 2.0 API non-blocking interface (twisted.enterprise.adbapi). 1) Simple database test

2) Simple transaction test

See also, RDB : Buildbot framework ORM : Twistar

[astinchoi]

Twisted] 7. Logging

1) File Logging

2) Error Logging

3) Rotating Logs old to new >> test.log.N to test.log.1, test.log

4) Custom logger

- twistd custom example $ twistd -ny echo_server.tac --logger=log_colorizer.logger --logfile=- 5) Bottom Line for logging - start logging : log.startLogging - event logging : log.msg (stdout) and log.err (stderr) - custom logger : use log.addObserver method  - when coding custom log observer, mustn't be block state, otherwise block entire event loop. should code in thread-safe. - twistd logging : enable auto logging. Can optimize --logfile, --syslog, --logger options

[astinchoi]

Twisted] 6. Deploy twisted application

Twisted is the engine which for scalable cross-platform network server and client.

1) Service

Something can start and stop and it implements IService interface

(TCP, FTP, HTTP, SSH, DNS, and many other protocols)

startService: Include reading setting data, setting database connection and listening a specific port

stopService: Include saving status at disk, closing database connection and stop listening a specific port

2) Application

is top-upper container with deploying single or multi service

Services register one application.

3) TAC File (Twisted Application Configuration)

Under twisted application, protocol implement body is a module. It register at application setting file(TAC). And reactor and setting managed by external utility.

It can convert echo server to echo application through simple algorithm

- move Protocol and Factory class to module.

- In TAC file, create twisted.application.service.Application instance, then register factory to service instead protocol factory to reactor. - echo.py

- echo_server.tac

4) twisted command-line utility

is cross platform utility to deploy twisted application, which are running TAC file and handling with start and end.

ex) $ twistd -y echo_server.tac

simply, start daemon instance of application, do logging at twistd.log and save PID at twistd.pid file. - twistd.log

please check $ twistd --help

we just got the daemon and logging function without any code.

5) Plug-in

is one of alternations to executing twisted application based TAC.

$ twistd echo --port=9999 6) Others - Current job directory list view $ twisted web --port 8080 path . then http://localhost:8080

- isolated DNS resolver 

hosts file facebook.com 127.0.0.1 then

- ssh at port 2222 $ sudo twistd conch -p tcp:2222

- execute ESMTP POP3(receive email and save emails directory for localhost domain) $ twistd mail -E -H localhost -d localhost=emails

- references  https://twistedmatrix.com/documents/current/core/howto/application.html https://twistedmatrix.com/documents/current/core/howto/plugin.html https://twistedmatrix.com/documents/current/core/howto/tap.html - TODO ch2 : chat server >> twistd application ch4 : non-blocking >> plug-in based service [Tip] - $ twistd --help ( is plugin) - can view the source at twisted/plugins directory and can read service definition

[astinchoi]

2015 계획

1. 몸이 건강하지 않고 컨디션이 나쁘면 일 하는데도 지장이 생겨서 건강을 우선으로 생각한다 ! 운동을 꾸준히 !! 그리고 적당히 먹자. 뷔페 같은데 가서 본전을 뽑아야 된다거나 욕심 부려서 많이 먹지 말자. 술도 될수 있으면 마시지 말자.

2. 조금 더 시간을 통제 해야겠다. 정해진 시간 내에 뭔가 안되면 과감히 내일 하자. 안되는데 시간만 붙잡고 있으면 효율성 뿐만이 아니라 다른 일에 지장을 주고 스트레스만 쌓인다. 조급해 하지말자.

3. 피아노, 기타 등 음악 하는 시간을 조금이나마 꾸준히 투자한다. 결과물은 녹화한 것으로 !!

4. 다니고 있는 회사에서 현재 하고 있는 일 + 알파를 한다. 올해 계획하고 있는 osx swift와 windows c#을 개발해서 사내 글로벌한 툴을 만든다. 회사에서 다양한 실험을 할 수 있는 환경이 다른 곳에 비해 기회가 많다. 회사의 솔루션 리소스를 이용해서 다양한 실험을 해보자 !

4. 전화영어 10분 2015 8개월간 예약되어 있다. 영어가 익숙해지면 중국어에 대해서 생각해보자.

5. 2015년도에 번역서가 출간 된다. 다른 번역서 혹은 집필에 도전해 보자. 그리고 하는 일이 개발은 아니지만 코딩에 손을 놓지 말자! 뭐라도 git 저장소에 올려 놓자 !

6. 올해 초까진 학자금과 생활비 대출을 청산 할 수 있다. 엄마 빚이 남아 있긴 한데.. 어떻게 도와 줄 수 있을지 조금 생각해 보자. 그리고 고시원 탈출 계획도 생각해 보자 ! 7. 규칙적인 생활을 유지하자. 불규칙적인 생활은 몸에도 안좋다. 그리고 한번 깨지면 유지하기 힘들다. 이런 경우가 있을때 다른 일들을 포기 해서하도 규칙적인 생활을 유지하는데 노력하자. - 당장 해야할 일 Variation of Kanon, River flows in you (Piano) 녹화 Twilight (Guitar) 녹화 WebTestTool(OSX, Windows) 두 버전 만들기 솔루션 실험용 웹 사이트 만들기 twisted 책 다 읽기 운동 (헬스 및 수영) 알아보기

[astinchoi]

Twisted] 5. Web client

1) Print web resource (for the little work)

2) Download Web resource (for the little work)

3) Agent(flexible, extendable) : Print web resource

4) Agent(flexible, extendable) : Print HTTP header

5) Agent(flexible, extendable) : Print Post Data - client example

- server example

6) Others.. - Agent API:  http://twistedmatrix.com/documents/current/web/howto/client.html - Example http://twistedmatrix.com/documents/current/web/examples/

[astinchoi]

Twisted] 4. Web Server

1) webecho with telnet

2) Parse HTTP Request

3) static URL dispatch

4) dynamic URL dispatch with redirect and isLeaf

5) POST request example

6) asynchronous response

7) Others - http://twistedmatrix.com/documents/current/web/howto/ - http://twistedmatrix.com/documents/current/web/howto/web-in-60/ - http://twistedmatrix.com/documents/current/web/examples/ - https://github.com/twisted/klein

[astinchoi]

Twisted] 3. Deferred

Callback is the way that reactor notifies application for event arrival. Deferred manages callbacks, which is beautiful abstraction.

terms are fire / callback / errback

Example 1)

output: <i><b>Hello World</b></i>

Example 2)

output: <h1>Breaking News: Twisted Takes us to the Moon!</h1>

output:  [Failure instance: Traceback (failure with no frames): <type 'unicode'>: The headline''123456789012345678901234567890123456789012345678901234567890'' is too long!]

Example 3)

#1.  Callback 1 said: Test Callback 2 said: Test #2. Callback 1 said: Test Callback 2 said: Test Unhandled error in Deferred: Unhandled Error ... exceptions.Exception: Callback 3

#3. Callback 1 said: Test Callback 2 said: Test Errback 3 took care of [Failure instance: Traceback: <type 'exceptions.Exception'>: Callback 3

#4. Errback 1 had an error on [Failure instance: Traceback (failure with no frames): <type 'unicode'>: Test] Unhandled error in Deferred: Unhandled Error Traceback (most recent call last): Failure: __builtin__.unicode: Test

#5. Errback 1 had an error on [Failure instance: Traceback (failure with no frames): <type 'unicode'>: Test] Errback 3 took care of [Failure instance: Traceback (failure with no frames): <type 'unicode'>: Test]

#6. Unhandled error in Deferred: Unhandled Error ... exceptions.Exception: Callback 2

#7. vs #3 Callback 1 said: Test Callback 2 said: Test Unhandled error in Deferred: Unhandled Error ... exceptions.Exception: Callback 3

#8. Errback 3 took care of [Failure instance: Traceback: <type 'exceptions.Exception'>: Callback 3 ... Callback 1 said: Everything is fine now.

bullet point for deferred 1) first, fire deferred. 2) deferred can fire only one, otherwise AlreadyCalledError 3) except deferred N'th >> handle N+1'th 4) N'th callback result will pass to 1st param of N+1'th callback 5) if errback receive non-failure object, it will be wrapping, if callback do except, it will be handling  addCallback addErrback: (catch) addCallbacks: same level with callback and errback addBoth: (finally)

[astinchoi]

Twisted] 2. Protocol State

There are two states: REGISTER, CHAT ChatFactory saves protocol state(dictionary which connected user) Don't do depending code at Protocol. The reason is to test with application and to reuse protocol easier. 1) http://twistedmatrix.com/documents/current/core/examples/ Twisted code example directory(UDP, SSL, and other server/client examples..)

2) http://twistedmatrix.com/documents/current/core/howto/ Twisted from scratch reference > http://autobahn.ws/python/ (Web socket)

3) http://twistedmatrix.com/documents/current/core/howto/endpoints.html Endpoints API (to provide high-profile server and client)

[astinchoi]

Twisted] 2. TCP Echo server/client more 1) Persistent protocol status is in Factory.

Also, it's same as.. # delete  __init__ method in QuoteProtocol class and .. class QuoteFactory(Factory):     numConnections=0     protocol = QuoteProtocol        def __init__ ...

2) Protocol can know why connection is finished.

3) Many client can make connections for one server. For this example, maybeStopReactor decides to what will do when connection end. It implicates a design issue in client.

[astinchoi]

Twisted] 1. TCP Echo server/client Twisted: Event-driven Networking Engene using Event Loop, Callback triggered by some action.

Single Thread vs Multi Thread vs Event Driven Single Thread: one job is stopped by I/O job, others should wait. Multi Thread: Efficient and need to take care for shared resource. Event Driven: make the place interleave for the job in one thread flow. Expensive calculations such as I/O job will be added to event loop with callback. Event Loop is polling recursively that event is done. Once event is done, it will send it through call back.

1. reactor : is Even Loop, can distinguish Network, File system, Timer Event and wait event and demultiplex it and send it to event handler. core behaviour is while True:     timeout=time_until_next_timed_event()     event=wait_for_event(timeout)     event+=timed_events_until(now())     for event in events:         event.process() 

briefly, reactor registers callbacks and starts event loop. 2. transport connect between end points. (TCP or UDP) (write, writeSequence, loseConnection, getPeer, getHost) 3. protocol asynchronous handling with http, telnet, dns, imap.. (makeConnection, connectionMade, dataReceived, connectionLost) 3.1. protocol factory The code creates instance of Echo protocol class for every connection. This means not saving persistent setting information So, this data keep in EchoFactory Class. Lastly, It's important to separate transport with protocol to reuse transport easily in twisted Design factor.

[astinchoi]

Go Source Install

Install for the source $ tar -C /usr/local -xzf go$VERSION.$OS-$ARCH.tar.gz $ export PATH=$PATH:/usr/local/go/bin $ cd go/src $ ./all.bash

[astinchoi]

LCL] 36. Last chapter, other tools.

1) Redirection  Group Command vs Subshell Command

Group Command: runs in current shell Subshell Command: runs in child shell (copied by current) 2) Substitute process Pipeline command always runs in subshell. so you can substitute process as 2 ways below. creating std-output: <(command-list) getting std-input: >(command-list)

--

3) Trap (See also LCL] 10. Signal: http://astinchoi.tumblr.com/post/90342467958/lcl-10-process)

usage: trap argument signal [signal...]

Temporary(Tmp) files. One of reasons script includes signal handler is to remove temporary files. Also, the reason it needs temporary files is to share with other processes. So, It is important to make temporary file name not expectable.  you can make like this..  1] tempfile=/tmp/$(basename $0).$$.$RANDOM but RANDOM shell variable makes 1 to 32767. so you can use mktemp program 2] tempfile=$(mktemp /tmp/foobar.$$.XXXXXXXXXXX) like this /tmp/foobar.6593.sDFew02382Sdfde...

General user recommend to make temporary not in /tmp, but $HOME/tmp. [[ -d $HMOE/tmp || mkdir $HOME/tmp

4) Asynchronous Execute  Wait command

5) Named Pipes to use connecting two processes. Communicate processes such as client/server network connection. It does like file, but FIFO buffer form. you can set as below.. process1 > named_pipe and process2 < named_pipe it do as.. process1 | process2 Example) p is named pipe

- Terminal1

(now blocking) - Terminal2

- Terminal 1 again

(exit block, because of FIFO, terminal2 takes it from terminal1)

[astinchoi]

LCL] 35. Array

1) hours

2) array operation

3) array operation 

[astinchoi]

LCL] 34. String and number

1) Parameter extension

//: all #: from first %: from last

2) time

3) $(( expression ))

4) $ bc Shell script can't calculate well for float number. So when you want to do so, you may use other ways. bc is one of those.

[astinchoi]

LCL] 33. For Loop

1) single

2) multi

3) C style for statement

4) Example