shalala66 - Tumblr blog

shalala66 · 4 years ago

Photo

“CS” --> “AI” --> "CV" sahesinde "Uz tanima" texnikalari haqqinda

Uzun muddetdir uzerinde ishlediyim "Komputer Gormesi"nin daha hessas ve bir o qeder chox zehmet, eziyyet teleb eden "Uz tanima" layihemin proqram teminati ile bagli davamli shekilde, hemchinin, muxtelif variasiyalarda proqram teminatlari, eynile, nezeri-populistik (p.s.: eynile praktiki) meqale(-ler) hazirlamagi qerara aldim.

"Uz taninmasi"nda birinci etaplardan en esasi insan(-larin) uzunun shekil ve ya videoda, elece de real-zamanda kamera vasitesile elde edilen kadrlardan mueyyen edilmesidir. "Face detection" sonrasi icra olunan prosedur "Localization"dir. Hansi ki, "detect" edilen uzun yerleshdiyi koordinatlar ve uzun olchusu nezere alinir. Tarixi meqam 47 illik omrunde chox boyuk texniki ishin esasini qoyan, bu gunku "Suni Intellekt" texnoloji bazarinda muracietler edilen kaskadlar bilavasite Macaristan esilli boyuk riyaziyyatchi-alim Alfred Haar-in ismi ile baglidir,- Haar olchusu, Haar dalgasi ve Haar funksiyalarinin ortoqonal sistemi kimi murekkeb movzularin banisi kimi de taninir elm dunyasinda. Bes, "ortoqonal sistem" riyaziyyatda ne demekdir?! Vektorial elementlerin skalyar hasilidir. {fi} c H- alt vektorlarin skalyar hasili sifira beraberdirse, ortoqonal adlanir. (fi, fi) = 0 Alfred-in riyaziyyatda esas tetbiq sahelerini de esas etibarile ortonormal/ortoqonal sistem funksiyalari, analitik funksiyalar, differensial tenlikler ve s. ehate edib. Qeribedir ki, doktorluq dissertasiyasini Hilbert-in rehberliyi altinda etmesine baxmayaraq, Fon Neyman ve Pontryagin(Lev Semenovich) Hilbertin beshinci probleminin arashdirilmasinda adindan belli oldugu kimi Haar'in ozune istinad ederek, "invariant Haar olchusu"nden faydalaniblar. Arashdirma ise "topoloji qrup"lari oyrenmeye qulluq edirdi. Riyaziyyatda topologiyaya esas 2 yanashma ayrilir: 1. Umumi anlamda - davamliligin, kesilmezliyin erseye gelmesi. 2. Ayri(ca)liqda - davamli deformasiyalar neticesinde oz veziyyetini saxlayan xasseler.

"μ-, μ'- Haar olchusu" riyaziyyatda daha chox "Sag/sol Haar olchusu" kimi taninir. Chox qisa olaraq, "Sol Haar olchusu" movcud sherti "G- (Haussdorf- un lokal komponent topoloji qrupu)" qrupunda "σ- chevrede" mueyyen edilmish "μ- olchudur" (Borel -in 0-dan ferqli reqular olchusu): µ(gA) = µ(A) g ∈ G, A ∈ G. "Sag Haar olchusu" ise eksinedir: μ'(A) = μ'(A^-1) ==> µ'(Ag) = µ'((Ag)^-1) mueyyen edilmish "μ'- olchudur".

"Haar dalgasi(wavelet)" verilenlerin ferqli tezlik komponentlerini analiz etmek uchun yuxarida qeyd etdiyim "ortoqonal sistem funksiya"si uzerinde qurulan riyazi funksiyadir.

"Haar funksiyasi" [0, 1) intervalinda mueyyen edilen funksiyadir. "Riemann inteqrali" ile [0, 1) araliqinda inteqrallanan her bir funksiyani Haar funksiyasi ile siralamaq (siralara parchalamaq, bolmek) olar. Bu "Haar sirasi":

adlanir.

"Haar sirasi" natural ededler uchun:

Eynile "Furye sirasi" kimi:

Yeri gelmishken, "Riemann" hipotezi de "CMI- CLAY Mathematics Institute" terefinden 2000-ci ilde teyin edilen 100 illik hell edilmemish "7 riyazi problem" sirasina daxildir. Onlardan yalniz biri, yeddinci problem("Poincare" hipotezi) XX esrin en parlaq riyaziyyatchilarindan belke de en birincisi, "The Guardian" neshrinin dercine gore gelmish-kechmish 10 en mohteshem "genius" riyaziyyatchilar sirasinda qerarlashan "Grigori Perelman" terefinden hell edilmishdir (https://www.theguardian.com/culture/2010/apr/11/the-10-best-mathematicians).

O meqama da toxunmaq lazimdir ki, Perelman-in riyaziyyat konfransinda teqdim edilmesi planlashdirilan 1 milyon pul mukafatini redd etmesine esas sebeb daha evveller bu problemin helli ile bagli tekliflerde bulunan "Richard Hamilton"a qarshi edilen haqsizliq, sayqisizliq olmushdur. Bir sozle, Tyurinden sonra Hamilton terefinden bashladilmish arashdirma Perelman terefinden sonlandirilmishdir. Qeribe meqam ise onadan ibaretdir ki, cemi 1 il sonra Hamilton sherqin riyaziyyatchilar uchun "Nobel mukafati" adlandirilan "Show Prize" mukafati ile teltif edilir. Qrisha-nin atdigi redd addimindan ferqli olaraq, miqdari 1 milyon dollar teshkil eden mukafat Hamilton terefinden qebul edilir. Bashqa bir maraqli meqam odur ki, Perelman problemin hellini aidiyyati quruma teqdim etdikden sonra bir kimse onun dogrulugunu yoxlaya bilmir. Bu ise, problemin demek olar ki, riyaziyyatin bir chox qollarini ehate etmesi idi. Neticede bir rus riyaziyyatchisinin teklif etdiyi helli diger boyuk alim, "Abel" mukafati laureati, rus esilli amerikan ve fransiz riyaziyyatchisi "Mixail Qromov" tesdiq edir. Ashagi-yuxari bu movzu ile chox az adam tanishdir. Ancaq, men eminlikle deye bilerem ki, ekseriyyet insanlar meseleni umumiyyetle anlamir, digerleri ise tamamile sehv basha dushurler. Chunki, Perelman yalniz "Poincare hipotezi"nin hellini vermishdi. Lakin, "duz (hamar) dordolchulu Poincare ferziyyesi" hele de hellini tapmayib. Yeni, dordolchulu topoloji sferanin iki ve ya daha chox qeyri-beraber duz (hamar) qurulusha malik olub-olmamasi problemi hele de "unsolved" statusunu dashiyir (red. CMI- Smooth 4-dimensional Poincare conjecture). Umumiyyetle, Puankare hipotezi 1D mustevi de daxil olmaqla, 2D (red. Puankarenin ozu terefinden isbat edilib), 3D (Perelman terefinden isbat edilib), 4D (yuxarida qeyd etdiyim kimi hell edilmeyib) topoloji problemidir.

Xulase, Haar-in riyazi alqoritmik metodunu bir az daha tekmilleshdirerek, Viola ve Johns 2001-ci ilde komputer texnikasi elmine oz tohvelerini verdiler. Haar metodunda oldugu kimi insan uzunden savayi muxtelif obyektlerin ashkarlanmasinda da istifadeye yarayirdi. Insan uzunu misal chekesi olsaq, duzbucaqli sahelerden ibaret piksellerin cemlenmesi emeliyyati hesabina pikseller regionunun insan uzunu teshkil etmesi mueyyen edilir. Bezi texniki edebiyyatlarda "ortogonal ve ya rectangular filter" kimi adlanir. Daha deqiq ve tez bir zamana netice veren "steerable filter" istifade edile biler. Lakin, ortoqonal filterlerin esas ustunluyu "processing" zamani shekilleri inteqral formatda saxlamasidir. Ki, "detection's time complexity O(1) const time"a yerine yetirilir. Inteqral format ise, "integral image" adlanir:

Bu, o demekdir ki, shekli teshkil eden her bir piksel ozunden solda ve yuxarida yerleshen qonshu piksellerin cemine beraberdir. Qonshu piksellerin etrafinda

yerleshen, bir-birine mesafesi texmini 1-2 piksel teshkil eden pikseller ile girish sheklinin AxA olchulu sag ashagi kunc matriksini elde edirik ki, bu da "wanted area"dir. Bu texnikani ilk defe "Frank Crow" "Komputer Qrafika"sina daxil edib. Sonradan, "Lewis" terefinden "Computer Vision"a da transfer edilib. Shekilde ardicilliqla "Edge, Line ve Four-rectangle feature"lar gosterilib:

p.s.: Frank C. Crow ve Lewis’in resmlerini hech bir internet menbeden elde ede bilmedim.

"Background"da Viola & Johns metodu "Ada Boost (Adaptive Boosting) by Freund & Schapire" adli mashin oyrenmesi alqoritmi tetbiq edile biler. "Ada Boost" obyekte gore klassifikasiya aparir. Ve, bu klassifikatorlari bir nov "merge" edir. Qisaca onu da bildirim ki, klassifikatorlarin(red. bu "unity comitet" deye qeyd olunub) ish prinsipi her obyekte gore daha evvelki prosesde sehv klassifikasiya edilenlere hesablanib. Tebii ki, esas ish prinsipi axtarilan obyektin (ROI- Region of Interest) pozitiv, eks obyektler sinfinin ise neqativ qruplashdirilmasina dayanir. Artan murekkeb iyerarxiya ile orqanize edilir. Ki, bu da oz novbesinde "cascade" adlanir. Lazimi obyektin tapilmasi insan uzu nahiyelerinin bolgusu(face labelling) ile heyata kechirilir. Bele ki, burunun mueyyen olunmasi uchun vertikal, eksine gozlerin tapilmasi uchun goz etrafi halqalarin insan uzunun diger regionlarina nisbeten daha tund ve qaraya chalan olmasina gore horizontal duzbucaqli filterlerden istifade edilir. Minimal "error rate"e sahib konsept "Haar cascades"dir. Uzun mueyyen olunmasina qulluq eden "cascade"lardan savayi muxtelif obyektlerin de mueyyen olunmasi uchun "Haar cascade"lar (XML formatda) movcuddur. Butun bu emeliyyatlar,- uzun mueyyen olunmasi, esasen 2 etapa yerine yetirilir:

1. Characteristics - AxB mashtabi. statik ishiqlandirilmish muhit. muxtelif istiqametlerde ve bucaqda ferqli uz ifadeleri.

2. Preprocessing - cropping -> butov goruntuden lazim olan hissesin alinmasi. grayscale converting -> diger konvertasiyalar(BGR2RGB) da tetbiq edile bilinir. Bu ise, yaxin kechmishde sahibkarlarin BGR standarti ile mehsul satishina dayanir. resizing -> standart teyin etdiyimiz olchunun elde edilmesi. equalization -> daha dogrusu, "histogram equal." ferqli ishiqlandirilmaya malik zonalarin "smoothing"i(beraberliyin elde edilmesi) uchun istifade edilir. filtering -> "billaterial filtering"(2x) kichik detallarin "smoothing"i uchun muraciet edilir.

Vurqulamaq lazimdir ki, umumileshdirilmish halda uz tanima emeliyyati da 2 etapa sonlanir: 1. Training. 2. Recognition.

Bu hisseye qeder behs etdiklerim xususiyyetler sayesinde klassik uz tanima metodunu(via CV), bir sozle, kaskad klassifikatorlarini ehate edir. OpenCV GitHub layihesinin bir hissesi oldugu uchun de bashlica metod "multi-scale detection"dir(ozunde uzun tapilmasi uchun shekil uzre ashagi-yuxari hereket sayinin teyinine gore "scale factor (by default 1,1)", uzun mueyyen olunmasindan deqiq emin olmaq uchun kifayet edecek duzbucaqli sahelerin sayi uchun "minimal neighbors(by default 3)", oxunan fraqmenti cemleyir). Tebii ki, "manual" mudaxile chox teleb olunur. Daha neologist usul ise "derin oyrenme" ile MTCNN (by Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, Yu Qiao) vasitesile uzun tapilmasi, mueyyen edilmesidir. Adindan gorsendiyi kimi DL (deep learning) ile kolvulsional neyronal shebeke sayesinde qurulur. Elave ferqli ceheti "facial landmark"dir:

MTCNN piramidal bolguden sonra sub-net olaraq, ozunde 3 CNN birleshdirir: (P)roposal-Net -> uz sahesi(facial region), (R)efine-Net -> serhedleyici cherchive(bounding box), (O)utput-Net -> uz nahiyeleri(facial landmarks). Ona gore de "Multi-task Cascaded Convolutional [Neural] Networks" ismini dashiyir. "DL framework - Caffe" proyekti cherchivesinde "release" edilib. "Tensorflow/Theano/TFLearn-Keras/Caffe kimi framework"larin nezninde istifade edilir. Muhum rol oynayan metod mtcnn()-dir. Ozunde yuxarida adi kechen "scale factor", "box - x, y (height, width)", "[prediction] confidence" ve ferqli olaraq, "keypoints (face labeling -> right/left -> eye)" cemleyir:

Yuxarida eks etdirilen "MTCNN's local landmarks" texnikasidir. "General landmarks model" ise esasen "labeling", "blinking", uz simmetriyasinin, veziyyeyinin mueyyenleshdirilmesinde istifade edilir. Numunedeki 68-d annotasiyadan ferqli 128-d, 194-d modeller de movcuddur.

Bu hissede, yene de onceki meqalelerden birinde chox genish toxundugum her uch "sub-net CNN"de "NMS (Non-max suppression) & BB(Bounding Box)" ile IoU (Intersection over Union) alqoritmi ishleyir:

Bes, niye NMS IoU sayesinde ugurlu performans sergileye bilir?! Cavabi chox sadedir.. Ona gore ki, obyekt ashkarlanmasi elminde esas problem "mueyyenleshdirme, klassifikasiya ve lokalizasiya" proseslerinin bir neche ve yaxud her matriks, yeni, "grid of cell"e gore ferqli "centroid"lerle qeyd etmesidir. Ki, "Non-max suppression" "IoU" vasitesile en yuksek ehtimalliq deyerini(probability value) tapmaqla, optimizasiya tetbiq ede bilir. Bele deyek, bu elmde "predicted box ve ground truth" movhumlari var. IoU (by default > 0.5) bu iki movhumu birleshdirmeye xidmet edir:

Tam terifi veresi olsaq, ehtimalliq "Sherti Ehtimalliqa(Conditional Probability)"a ehtiva edir. Bu Bayes teoremidir:

ve ashagidaki shekillerden gorunduyu kimi dusturun mexrecinde "Mutleq Ehtimalliq" hadisesi de yer alir,

hemchinin, Venn diaqramina da muraciet eks edilir (red. Euler diaqrami da elaqeli movzudur):

Indi ise, uz tanima ile bagli tarixi-nezeri-praktiki meqamlara toxunmaq isteyirem. Avtomatlashdirilmish (daha dogrusu yari-avtomatlashdirilmish) uz tanima sisteminin pionerleri hele 1960-ci illerden aidiyyati sahede ugurlar elde eden "Helen Chan Wolf (dunyada ilk avtonom robotun muellifi),

Woodrow Wilson Bledsoe"

ve neinki her hansi bir shekli, hemchinin, "Wikipedia" da daxil olmaqla hech bir internet sehifesinde haqqinda normal bilgi olmayan "Charles Bisson"dur. Bildiyimiz kimi insanlar uchun bu tapshiriq resmi dokumentasiyada da vurqulandiqi kimi ele de chetin deyil. Insanlarda lap kichik yashlarindan uz tanima qabiliyyeti formalashmaga bashlayir. Beyinimiz qarshimizda canlanan goruntunu nece analiz edir ve kodlashdirir? Bu barede ilk elmi ish felsefe ve tibb elmleri sahesinde Nobel mukafati laureatlari, "neuroscience"da evezsiz xidmetleri olan chox boyuk alimler "David Hunter Hubel

ve Torsten Nils Wiesel"

terefinden beynin xett, bucaq, siluet, kenar hisseler, elexsus, herekete reaksiya veren lokal, daxili spesifik xususiyyetlere malik xususi sinir huceyrelerinden ibaret oldugu teqdim olunub. Bele ki, biz etraf-alemi hisselere parchalanmish halda deyil de, beyin qabiqi sayesinde birleshmish shablon formasinda goruruk. Bes, komputerler nece? Onlar uchun prosesin murekkeblik derecesi ne qederdir? Eynile, insan beyninde oldugu qeder. Sadece, insan beyni her daim "backend"de ishlek oldugu uchun bir chox ishleri bilavasite prosese biz terefden mudaxile olunmadan, gorur. Uz taninmasinda intuitiv yanashmalardan(Evklid mesafesi ile insan uzunun hendesi qurulushunun etalon sheklin vektorial xususiyyetlerine gore hesablanmasi - gozlerin, qulaqlarin, burnun movqeyi ve s.):

tutmush daha muasir( [Eigenfaces] - choxolchulu shekil mustevisinden ibaret noqte. AxB olchulu sheklin grayscale chalarina kechidde S=AxB 3D vektorial muhiti zebt etmesi, 100x100 pikselli sheklin 10.000 pikselli 3 olchulu muhitde yerleshmesi yuksek choxolchululuk problemi yaradir, Karl Pearson

ve Harold Hotelling

PCA hellini teklif edibler. Meqsed korelyasiyaya meruz qalan "data"ni minimuma endirmek idi,

[Fisherfaces] - Eigenfaces'in "kernel"i PCA uzerinde qurulub(red. PCA - Principal Component Analysis vasitesile verilenlerin dispersiyasini maksimallashdiran funksiyalarin xetti kombinasiyasi tapilir, lakin, dispersiya butun sinif verilenlere tetbiq edildiyi uchun "data info" itkilerlerle bagli problemler meydana chixir). Adindan melum oldugu kimi statistika muhendisi Ronal Aylmer Fisher

terefinden daxil edilib,- LDA - Linear Discriminant Analysis(xetti diskriminant),

[LBP 2^8 (0, 1) kombinasiya -> LBPH - Local Binary Patterns Histograms] - 2D textural analyze) ve murekkeb "solution"lar istifade edilmekdedir. Umumi qruplashdirma aparsaq, "geometry(SVM, LDA, PCA <=> LPP <=> ICA, DCT, Gabor, ), piecemal(HMM), appearance/model(Kernel PCA), statistical(NN, NN + Gabor, NN + HMM, NN + Fuzzy) based") kimi usullara qeder bole bilerik.

-----------------------------------------------------------------------------------------------------------

neyropsixoloji neyrofizioloji informativ-prosedural komputer modelleri

1 note · View note

shalala66 · 5 years ago

Text

Characteristics of Class, Instance, Object in Python

Python proqramlashdirma dilinde Class'in instance'i, yeni, object'i (qeyd: Mehz Class instance ve Class object Python proqramlashdirma dilinde umumi object yanashmasini nezere almadan, chox cuzi anlamda ferqli menalar kesb edir. Class ozluyunde bir konsept prototipdir, Class object data abstraksiyadir ve real Class copy'dir, hansi ki, main memory'de space allocate edir, Class instance virtual Class copy'dir ve Class'a referance'dir, hansi ki, __new__ method'u vasitesile referance edir.) yaradilanda, ilk novbede

__new__(creator) method-u chagirilir:

1. Onun uchun memory-de area ayrilir.

2. Bu object-in lib-leri, method-lari, __dict__-leri yaradilir.

3. Object-in referance deyeri artirilir.

Elave: __new__ - Class-in static method'dudur. Class-in ozunu temsil edir ve cls.cls parametr-ini teleb edir. Compiler, Class-in (exemplar or instance)-ini yarananda, bu parametri teqdim edir eslinde. Daha deqiq, class chagirilanda, __new__ method-u da chagirilir o an. Python'da __new__ method'unun rolunun yaxshi anlamaqdan otru Java'da bir kod fraqmente nezer salaq:

public class Base { int x = 7;

public static void main(String[] args) { Base myObj = new Base(); System.out.println(myObj.x); } }

Java'nin sintaksisinde explicitly new operatoru, Python'da implicitly __new__ method'u esas rol oynayir.

Eger, class daxilinde __new__ method-u varsa, o __init__ ve diger method-larin calling convection-unu, chagirilish ve ya eksini mueyyen edir. Class-in examplar ve ya instance-ini ele Class-in oz daxilinde super function istifade ederek: instance = super(Superclass, cls).__new__(cls, *args, **kwargs), ya da __new__ method-una direct muraciet ile: instance = object.__new__(cls, *args, **kwargs) create etmek olar.

__new__ minimum 1 argument qebul edir. Bu ise, class-in ozu olur.

Meselen: def __new__(cls, *args, **kwargs): return object.__new__(cls) cls - __init__ method-undaki self kimi parametrdir. Class-in ozunu bildirir.

Python2.2 versiyasina qeder (bu dovr old style'i ehate edir) esasen User defined ve Builtins class'lar ferqlilik teshkil edirdi. Python2.2-den bashlayaraq "object" daxil edilir (bu, new style dovrunun bashlangici idi). Pythonx.x < 2.2-e kimi old style bele idi - class Superclass:, new style ise object getirildiyine gore: class Superclass(object) formasinda. Python2.2-den new style-a daxil idi - 1) Descriptors support: 1) property - set, get, delete, 2) __slots__, 3) staticmethod, classmethod. 2) __new__ - static builtins method, 3) MRO(Method Resolution Order), C3(Linerization), LPO(Local Presedence Order), 4) super - builtins method.

class T(): pass

t = T() print(t)

class T1(object): pass

t1 = T1() print(t1)

Netice: <__main__.T instance at 0x7f9b196505f0> <__main__.T1 object at 0x7f9b1964d790>

Python3-de ise qismen ferqli. Bele ki, Python3-de object-den inheritance-i explicitly qeyd etmeye ehtiyyac yoxdur: class Superclass(object) qeyd edilir, eger,- proyektde Python2 ile uzlashmaya ehtiyyac olacaqsa. Yox eger, proyekt yalniz Python3 uzerinden reallashacaqsa, onda, class Superclass: formasinda da yazila biler. Zaten, Python3-de class Superclass = class Superclass(object):

class T(): pass

t = T() print(t)

class T1(object): pass

t1 = T1() print(t1)

Netice: <__main__.T object at 0x7f0d8c751080> <__main__.T1 object at 0x7f0d8c751128>

object - Python ecosystem-inin fundamentlerinden biridir.

Xususile vurgulayim ki, "her nesne obyektdir" meqami yuxarida xususi qeyd etdiyim Python2.2 fakti ile baglidir. Yeni, her nesne object class-indan inherited olur. Ve, __new__ bu object class-inin method-udur. Buna gore de, bu metod ile yuxarida sadaladigim 3 prosesi icra ede bilirik. __new__ ise ozu de ashagi seviyyeli (low-level) function-lari chagira bilir deye, bu ishleri gore bilir. Sirf bu __new__

method-u ile object class-inin elementleri yaradilir, oturulur ve sonda geri qaytarilir. Ve, mehz onun sayesinde class-in ancaq istenilen sayda instance-i yaradila bilinir. Yeni, limit tetbiq edile bilinir (p.s.: dolayisi ile Singleton pattern'i xatirladir, ancaq 1 instance-dan daha chox teyin etme furseti verir). Ve yaxud, mumkundur ki, bir neche instance eyni yaddash sahesine referance etsin. Ve ya, bir neche instance ferqli yaddash sahelerine referance etsin.

*args(primitive types, list, tuple) - positional (non-keyword) argument-leri, **kwargs(dict) - keyword (named de adlandirilir) argument-leri qebul edir.

Sonra, __init__ method-u (constructor) call olunur.

Ardi ile argument-ler bu method-a oturulur. Class-in (exemplar or instance)-i initialize edilende,

bu method chagirilir. Demeli, class name harda call olunsa, __new__ method-u da call edilir. __init__ method-u ise, __new__ terefinden class-in instance (exemplar)-i return olunaraq,

__init__ method-una self parameter kimi send olunanda, call edilir. __new__-daki "super function" (qeyd. super() - Bes, super() nece ishleyir? Builtins function olan super() Base class-in method-larina access almaq uchun, Base class(Superclass)-in temporary, proxy object-ini return edir. Bundan elave, Base class(Superclass) name-inin explicitly istifade edilmesinin qarshisini alir. Daha chox ashagilarda da qeyd edildiyi kimi polymorphism ve multi-inheritace zamani istifade edilir.) olmadan, __init__ ishe dushmur. Hem __new__, hem de __init__ return edirse, __init__-de error generate edilecek. Chunki, __init__-in vezifesi return etmek deyil de, Class-in yeni yaranmish instance(examplar)-inin cari veziyyetini, state-ini initialize aninda deyishmekden ibaretdir:

class InstanceQuant(object): _instances = [] limit = 3

def __new__(cls, *args): print("cls") if not len(cls._instances) < cls.limit: raise RuntimeError instance = object.__new__(cls) cls._instances.append(instance) # return super(InstanceQuant, cls).__new__(cls) # normalda - super() - # istifade return instance # edilir. Amma, burda # error generate edir. # Chunki, class-in ozunu # return edir, _instance # yox. def __init__(self, r): self.r = r print("self")

def __del__(self): self._instances.remove(self)

a = InstanceQuant(5) b = InstanceQuant(3) c = InstanceQuant(2) d = InstanceQuant(1)

__init__-in initializer method oldugunu "yield" istifade etmekle de, numayish etdirmek olar. Chunki, "yield" bu method'u generatora chevirir. Chox xirda kod numunesinde error generate olunacaq:

class Gen(object): def __init__(self, i): yield i

# Gen(5) g = Gen(5)

Error: Traceback (most recent call last): File "main.py", line 5, in <module> Gen(5) TypeError: __init__() should return None, not 'generator'

En sonda __del__ method-u (destructor) icra olunur.

Umumi metodlara nezer salaq: Construction(creation) & Destruction Compairing Unary

Math: __add__ Math(opposite - ters riyazi): __iadd__ Type change Format

Call: __call__

Context manager: __exit__, __enter__ (red. Elexsus, Context mexanizmleri OS internal proseslerin idare edilmesinde muhum movqe tutur)

Container: __getitem__, __setitem__, __delitem__ Attribute controller: __getattr__, __setattr__, __delattr__

Tebii ki, umumi "object" movzusu Python'da chox muhum, chox daha genish ve chox deqiq analize ehtiyyac duyur.

Esas 3 xasseye:

1) Referance count (for memory management)

2) Type

3) Value

ve, 2 hisseye bolunur: 1) Mutable (list, set, dict, byte array) 2) Immutable (int, float, str, bool, complex, frozenset, tuple) - primitive types (red. bu hisse Java dilinde tamam ferqlidir)

Mutable ve ya immutable yoxlamaq uchun id() ve yaxud is (iki obyektin yaddashdaki location'u check edir) keyword'unden istifade etmishem. int (integer) immutable (deyishmez) oldugu uchun x deyishenine yeni menimsedilen value yeni yaddash erazisine yazilir:

>>> x = 5 >>> id(x) 1704427760 (address) >>> >>> >>> x += 1 >>> id(x) 1704427792 (address) >>>

Yene de, "int type"-da oldugu kimi "str type" da modifikasiya olunanda, yeni object yaradilir ve yeni yaddash erazisine locate olunur:

>>> s = "Tural" >>> id(s) 2131050163424 >>> >>> >>> s += "Jalilov" >>> id(s) 2131050185008 >>>

Bele bir sual ortaya chixir.. Bes, nece olur ki, list kimi obyektler modifikasiya olunsa bele, locate edildiyi yaddash erazisi deyishmez qalir? Chunki, "list" obyekti background'da "+=" emeliyyatini, yeni, builtins math __add__() method'unu __iadd()__ method'u ile evez edir:

__add__(): >>> a = 10 >>> >>> a, b = 10, 11 >>> a.__add__(b) 21 >>>

__iadd__(): >>> l = [1, 2, 3] >>> id(l) >>> 140614678512200

>>> l.append(4) >>> id(l) >>> 140614678512200

Bu, combine method'dur, hansi ki, method'un self parametrini modifikasiya edir ve eyni id'ni return edir. "str, int" ve s. object'ler __iadd__() method'una sahib olmurlar, __add__() => iadd() convert ede bilmirler deye, immutable object'ler adlanirlar. Daha aydin shekilde, primitive type'lara yeni deyer menimsedilende, onlar modifikasiya olunmaq evezine yeni primitive type object yaradilir. Amma, maraqli bir meqam var burda. Python interpretator olan CPython'da -5 ve 256 intervalinda yerleshen integer object'ler Singleton xarakterli olduqlari uchun bele object'lere initialize olunduqlari value'nu yeniden menimsedende, yeni object yaranmir ve ilkin memory'deki location saxlanilir. Singleton olmayan object'lere ise eyni bir value assignment etdikde, memory location'u deyisherek, yeni object create olunur:

>>> x = 5 >>> hex(id(x)) '0x659780f0' >>> >>> x = 5 >>> hex(id(x)) '0x659780f0' >>> >>> >>> y = -3 >>> hex(id(y)) '0x65977ff0' >>> >>> y = -3 >>> hex(id(y)) '0x65977ff0' >>> >>> >>> z = 256 >>> hex(id(z)) '0x6597a050' >>> >>> z = 256 >>> hex(id(z)) '0x6597a050' >>> >>> >>> d = 257 >>> hex(id(d)) '0x1db7d75bef0' # ilkin address >>> >>> d = 257 >>> hex(id(d)) '0x1db7da40310' # sonraki address >>>

Mutable & Immutable object’ler..

Python dilinde Immutable object'leri Mutable object'ler kimi simulyasiya etmek istesek, onlari pointer'ler kimi istifade etmeliyik. Pointer'ler ise Python'da birbasha olaraq, movcud deyiller. Onlari sadece bir neche yolla imitasiya ede ve ya birbash C dilinde yazilmish proqramdan CPython (ctypes) lib'i ile chagira bilerik. ctypes tipli deyishenler declare etmek lazimdir. Ve, "pass by value" ile object'in copy'si uzerinden deyil, real object'in ozu ile emeliyyat aparmaq uhcun "pass by referance" istifade edirem. Bu proseslere deep-dive etmek bir az pedantik gele biler, ancaq chox vacib biliklerdir:

C dilinde:

#include <stdio.h>

void simulate(int *ch) { *ch = 502; }

int main(int argc, char **argv){ int var = 500; printf("var = %d\naddress = %p\n", var, &var); simulate(&var); printf("address = %p\nvar = %d\n", &var, var);

return 0; }

Results: var = 500 address = 0x7ffc1838563c address = 0x7ffc1838563c var = 502

Python dilinde: def simulate(ch): ch[0] = 502

var = [500] print(id(var)) simulate(var) print(var, var[0], id(var[0]), type(var), \ type(var[0]), id(var), sep='\n')

Result: 140600405082696 [502] 502 140600406331280 <class 'list'> <class 'int'> 140600405082696

Python'da C/C++'dan ferqli olaraq, deyishen yoxdur, deyishen adlari var.

1) C dilinde:

int x = 78; Address: 0x7f1 Value: 78

x += 1; Address: 0x7f1 Value: 79

int y = x; Address: 0x7f5 Value: 79

y += 1; Address: 0x7f5 Value: 80

1) Python'da: x = 78

Yuxaridaki simple assingment kodu erseye getirmek uchun background'da 6 etapdan kechir: 1) CPython --> PyObject creat edilir. 2) Bu PyObject uchun typecode int set olunur. 3) Bu PyObject uchun value 78 set olunur. 4) Maraqli meqam, name creat edilir - "x". 5) Bu "x" PyObject'e point edir. 6) PyObject'in referance count'u 1 vahid artir.

Python name (x) ----> PyObject (Type: Integer, Value: 78, Ref. count: 1)

struct PyObject.. Bu PyObject C-nin strukturu kimi mueyyen edilib. Ve, biz Python'da typecode ve ref. count'u call ede bilmirik. Chunki, Python'un C strukturlarina direct access'i yoxdur. Burdan aydin olur ki, Python'da declare/initialize etdiyimiz - "x" deyisheni background'daki PyObject'e point edir. Demek ki, "x" - referance'dir. Birmenali olaraq, Python'da bu "x" hech bir memory address'e direct olaraq, aid deyil. Chunki, PyObject CPython(C implementation)-a aiddir, CPython da oz novbesinde dolayisi ile C-ye aiddir. C-deki "x" ise "statical memory slot"a sahibdir. Python'da deyishene qiymet menimsetme yox, PyObject --> CPython --> C-ye referance var. Vessalam. 2) Python'da: x += 1 (yeni, x = 79)

1) new PyObject create edilir. 2) new PyObject uchun typecode int set olunur. 3) new PyObject uchun new value set olunur. 4) x new PyObject'e point edir. 5) new PyObject'in ref. count'u 1 vahid artir. 6) old PyObject'in ref. count'u 1 vahid azalir.

PyObject - OLD (Type: int, Value: 78, Ref. count: 1 => 0) Python name (x) ---> PyObject - NEW (Type: int, Value: 79, Ref. count: 0 => 1)

3) Python'da: y = x. Bu halda, y de x'in point etdiyi memory area'ya point edir.

Python name (y) ---> Python name (x) - PyObject (Type: int, Value: 78, Ref. count: 0) ---> PyObject (Type: int, Value: 79, Ref. count: 1 => 2)

>>> y is x >>> True >>> Ancaq, >>> x = 78 >>> y = 77 + 1 >>> y is x >>> True

1) 78 obyekti (PyObject) yaradilir (teyin edilir) 2) x name(x adi - string) 78 deyerine menimsedilir 3) 77 obyekti (PyObject) teyin edilir 4) 1 obyekti teyin edilir 5) 77 ve 1 obyektleri bir yerde add olunur 6) Yeni 78 obyekti teyin teyin edilir 7) y name (y adi - string) 78 deyerine menimsedilir

>>> a = 500 >>> b = 300 + 200 >>> False - bu hadise ancaq REPL-de bash verir. Ferqli IDE'lerde True return edilir. Ferqli IDE bu shertde hemchinin ferqli compiler demekdir. Eger, numune kodu .py file save edib, run etsek, True return olunacaq. Niye? ))))))) CPython compiler ve ya diger compiler'lar "smart compiler"lar qrupuna aiddir. Ve, koda optimization tetbiq edirler. Bu, "peephole optimization" adlanir, hansi ki, "execution step"leri (icra olunma pillerini) yadda saxlayir. Ele kod hisseleri var ki, hem REPL, hem de ferqli compiler'larda True return edir. Bu, "integrated (interned) object"lere goredir. Python "pre-created (onceden)" memory'de object'lerin sub-object (subset)'lerini yaradir ve onlari "global namespace"de saxlayir. Bu subset'lere daxildir: 1) Yuxarida toxunduqum -5 ve 256 araliqi ededler 2) Strings - "_", ASCII herfler, reqemler Bu memory allocating her defe bash verir. Menimce, "x is y" ve "a is b" neticelerinin sebebini yeterince inceledik. Umumiyyetle, "memory address compairing" uchun hashing de istifade etmek olar. Hecmhinin, sys.intern() ile hemishe eyni adrese muraciet etmek olar,- tebii ki, sonda eyni neticeni saxlayirlarsa.

4) Python'da: y += 1 (yeni, y = 80) beraberliyinde novbeleshme, muraciet ve icralarin testi.

PyObject - OLD (Type: int, Value: 78, Ref. count: 0) Python name (x) ---> PyObject - NEW (Type: int, Value: 79, Ref. count: 1) Python name (y) ---> PyObject - NEW (Type: int, Value: 80, Ref. count: 1)

Peephole optimization'un genish terifi:

Compiler-generated instruction'larin kichik set'ine optimizasiya texnikalarinin tetbiq edilmesidir.

Meselen: 1) 3 defe 8x8x8 evezine 8^3

2) a+a evezine left-right shifting

3) index'i 4-e vurub, base address'e elave edib, pointer dereferencing etmek evezine hardware addresing istifade edir.

Assembler level code fraqment'lere nezer yetirek..

No optimizer via 6 instructions:

a = b + c;

----------

MOV b, R0

ADD c, R0

MOV R0, a

d = a + e; ---------- MOV a, R0 ADD e, R0 MOV R0, d

Peephole optimizer via 5 instructions: a = b + c; d = a + e; ---------- MOV b, R0 ADD c, R0 MOV R0, a ADD e, R0 MOV R0, d

Hemchinin, eger compiler register'leri subprogram chagirilanda, stack'de saxlayir (yigir), sonra sondan evvele (LIFO prinsipi) boshaldirsa, bu, "reduntant stack instruction" adlanir. Peephole bu yigma-boshaltma (PUSH-POP) operation'larin tekrarlarini remove edir. Neticede, optimizasiya qazanilir.

Cpython’un Peephole optimizer’inin GitHub repo’sundaki source code’u (https://github.com/python/cpython/blob/master/Python/peephole.c):

/* Peephole optimizations for bytecode compiler. */

Qeyd: bytecode ve s. baresinde novbeti defe planladigim MMU movzusunda yazacagam..

0 notes

shalala66 · 5 years ago

Text

Bitwise & Constructor copy researching in C++

Choxdan test etdiyim xirda movzudur. Sadece, browser-e send etmishem. Function-a parameter kimi her hansi object gonderirem. Function qebul etdiyi original argument deyil, onun bitwise copy-si ile ishlemeye bashlayir. Process eynile, class constructor-da object yaradilanda, specific memory volume allocate olunur ve class destructor memory free icra edir.

Bitwise copy pointer-i bu minvalla original object address-i target edir. Function-la ish sona chatanda ve bitwise copy silinende, avtomatik olaraq, destructor ishe dushur ve memory free aparir. Tebii ki, original object-in locate olundugu memory area da free olur. Elave olaraq, general program sonlananda, destructor yeniden ishe dushur, hemin yaddash sahesini yene free etmek uchun time waste bash verir. Bu situasiyanin qarshisini almaq uchun constructor copy ile opposite tetbiq ede bilerik.

Object-in real copy-sini yaratmaq uchun esas meqam new memory allocate istifade edilir. Ki, initial assignment-e qeder empty oldugu uchun, ozunde ele memory address value saxlayir. 🤟🤟🤟

0 notes

shalala66 · 5 years ago

Text

OS braking overall, frozen booting of hdd and solution

Sistemde konkret OS braking overall-a gore, hdd boot ile hereket etmek mumkun olmayan hala gelmishdi. Ilkin olaraq, memtesting edildi ve malfunctions qeyde alinmadi.

Reanimator boot loading-le system disk check olundu, 200000 Mb-dan initial bad sector tapildi. Interval elde edilib, repair edildi.

Elave analizler olaraq, system file scanning prosesine bashlanildi. Sfc winlogon.exe notifications sayesinde ishleyir. Bezi virus ve diger malware kichik hecmli proqram teminatlari Winlogon notification packages elave edib, mask olunub, deyishiklikler apara bilerler. System registry-de hech bir elave file extension-a rast gelinmedi. Winlogon csrss.exe(p.s.: command prompt functions-da invoke olunur) ile ishe dushur. Her iki process non Win NT component smss.exe terefinden icra olunur. Bu servis boot loading process-den sonra WTM(task manager)-de yer ala biler. Kernel ve user mode-lari run etdiyi uchun, anlashilmazliq olsa, crash generate edir. Proses mecburi sonlandirilan hala gelse, bsod vere bilir. Bu hallar qeyde alinmadi. Sfc hansisa zedelenmish file uze cixartsa, dll.cache-den recover olunur. Protected resurslar uchun DACL ve ACL list-ler qurur. Bu listlere access TrustedInstaller.exe-den(CBS) olur.

CBS loader update/package level-de yerleshir ve CSI uchun komponentleri generate edir. Ashagiya dogru 4 level yerleshir. CBS.log-da bezi soft-larin sistemde install olub, sonra delete olunmasi tapildi. CBS CSI-i chalishdirir, CSI ise deployment leveldedir ve esas KTM olmaqla, DMI, CMI, SMI-i invoke edir.

Deleting zamani fayllar Component Storage-e gonderile biler. General scandan sonra cluster 103936-th index-de javaws.jar jdk18 file yerleshdiyi bad section tapilib, fix olundu ve diger zedeli sistem fayllari da ashkarlanib, berpa olundu. Ardicilliq POQ-a gore step by step icra olunur. P.S.: Bundan bashqa, physical monitor(lcd) problem - pixel damage ashkar olundu🤠🤘

0 notes

shalala66 · 5 years ago

Text

Data placing in a memory ile bagli eksperimentler

CPU data-lari memory-den oxumaq uchun her byte uchun one handling instruction icra etmir. Baxmayaraq ki, DMA kontrolleri CPU-nun produktivliyini artirmaq uchun routine operation-lari oz uzerine goturur. Bunun uchun diger solution-lardan context switching-i qeyd etmek olar. Adeten, stack-da sherti bir emeliyyatin saxlanilib, pre-operation yerine yetirmesi ve control-u o emeliyyata oturmesi.Bir az ferqli, diger process zero latency switching ile hech bir physical joining olmadan, diger kontentdeki operation frame-e pointer vasitesile qoshulur. Verilenlerin yaddashdan register-e oturulmesi GMA vasitesile bash verir. Processor MMX 64 bitlik 8 register xmm0 - xmm7-den SSE 128 bit 8 register xmm0 - xmm7(-xmm15) arxitekturasina kechid ederek, bir chox performans elde etmishdi. MMX hardasa simple integer, float emeliyyatlar(instruction[dest, src]) edirdi, cunki, register-leri soprocessor ile umumi idi. SSE-ye ise real digits soprocessor instructions-lar da elave olunub. CPU GMA vasitesile x86_32, x_86_64 architecture-den asili olaraq, 1-2-4 byte ve s. hecmde data-larla ishleye biler. 4 byte data 1, 2 set instructiona oxunulur. Esas mesele budur, datalarin memory-de siralanmasinda not aligned hallar bash verir ki, bu da, tebiidir. Bu siralanmalar oz novbesinde machine word-e gore boundary olunur. Type-dan asili olaraq, backward compatibility byte, half word, word, double word kimi bolunur. Tree iyerarxiyasinda bolguler de any, even, div by 4 ve 8 kimi qruplashdirilir. Load high - low byte bolgusu ve shift up - down-a gore, even address-de (1)2 emr, odd address-de (2)3 ve s. emre set olunur. CPU memory dovriyyesinde bandwidth, latency ozelliyi var. Bandwidth-i hardware hesabina rahat elde etmek olarsa, latency optinality-de situasiya bashqa curdur. Umumiyyetle, optimalliq elde etmek uchun reorganization of structures-a diqqet edilmeli(array-da mumkun qeder memory-cache access-in duzgun nizamlanmasina gore), suret uchun ise, temporary, spatial cache locality(hetta, extension olaraq cache line bouncing). Kod fraqmentleri uzerinde apardiqim sinaq testler esasinda bir sira suret ve cox cuzi optimalliqla CPU time neticeleri elde ede bildim:

0 notes

shalala66 · 5 years ago

Text

ML(Machine Learning) and AIN(Analyze of Intellectual Resources)

One of the dynamic methods of regression(linear, logical, robust, etc.) to find criterial variables via using predictors(regressors). It's been created by myself with the C++ codes to get result of core algorithm:

#include <iostream> #include <iomanip> #define half_s 20 #define half_ss 10

float train[half_s] = {1.1, 1.3, 1.5, 2.0, 2.2, 2.9, 3.0, 3.2, 3.2, 3.7, 3.9, 4.0, 4.0, 4.1, 4.5, 4.9, 5.1, 5.3, 5.9, 6.0}; float test[half_ss] = {6.8, 7.1, 7.9, 8.2, 8.7, 9.0, 9.5, 9.6, 10.3, 10.5};

float train1[half_s] = {39343.0, 46205.0, 37731.0, 43525.0, 39891.0, 56642.0, 60150.0, 54445.0, 64445.0, 57189.0, 63218.0, 55794.0, 56957.0, 57081.0, 61111.0, 67938.0, 66029.0, 83088.0, 81363.0, 93940.0};

float test1[half_ss] = {91738.0, 98273.0, 101302.0,113812.0, 109431.0, 105582.0,116969.0, 112635.0, 122391.0, 121872.00};

int main(int argc, char** argv) { //X/2 - train int i; float sum_train_x = 0.0; int count = 0;

for(i = 0; i < half_s; ++i) { sum_train_x += train[i]; ++count; }

//X/2 - train1 float sum_train_y = 0.0; for(i = 0; i < half_s; ++i) { sum_train_y += train1[i]; }

//X/2 - train * train float sum_train_x2 = 0.0;

for(i = 0; i < half_s; ++i) { sum_train_x2 += train[i] * train[i]; }

//X/2 - train * train1 float sum_train_xy = 0.0;

for(i = 0; i < half_s; ++i) { sum_train_xy += train[i] * train1[i]; }

//X/2 - test float sum_test_x = 0.0; int count1 = 0;

for(i = 0; i < half_ss; ++i) { sum_test_x += test[i]; ++count1; }

//X/2 - test1 float sum_test_y = 0.0; for(i = 0; i < half_ss; ++i) { sum_test_y += test1[i]; }

//X/2 - test * test float sum_test_x2 = 0.0; for(i = 0; i < half_ss; ++i) { sum_test_x2 += test[i] * test[i]; }

//X/2 - test * test1 float sum_test_xy = 0.0;

for(i = 0; i < half_ss; ++i) { sum_test_xy += test[i] * test1[i]; }

//Linear regression float a, b; float a1, b1;

//a = (sum_y * sum_2x - sum_x * sum_xy) / (count * sum_2x - sum_x * sum_x); a = (sum_train_y * sum_train_x2 - sum_train_x * sum_train_xy) / (count * sum_train_x2 - sum_train_x * sum_train_x); a1 = (sum_test_y * sum_test_x2 - sum_test_x * sum_test_xy) / (count1 * sum_test_x2 - sum_test_x * sum_test_x); std::cout << std::fixed; std::cout << std::setprecision(4) << "a = " << a << std::endl << "a1 = " << a1 << std::endl;

//b = (count * sum_xy - sum_x * sum_y) / (count * sum_2x - sum_x * sum_x); b = (count * sum_train_xy - sum_train_x * sum_train_y) / (count * sum_train_x2 - sum_train_x * sum_train_x); b1 = (count1 * sum_test_xy - sum_test_x * sum_test_y) / (count1 * sum_test_x2 - sum_test_x * sum_test_x); std::cout << std::fixed; std::cout << std::setprecision(4) << "b = " << b << std::endl << "b1 = " << b1 << std::endl;

float h, q; std::cout << "Enter the 'h' : "; std::cin >> h; q = b * h + a; std::cout << "q = " << q << std::endl;

float h1, q1; std::cout << "Enter the 'h1' : "; std::cin >> h1; q1 = b1 * h1 + a1; std::cout << "q1 = " << q1;

return 0; }

Then, extracted to SPDE(Spyder Python Development Enviroment - Anaconda Navigator) for a visualization response:

0 notes

shalala66 · 6 years ago

Text

GIL & Threading analiz

Global Interpreter Lock - tematikasinda genish researching ve testing

Deep dive-a varmaq uchun chalishdim ki, bu bolmeni ehate eden her nesneye toxunum. GIL esasen threading emeliyyatlarinda diqqet merkezinde oldugu uchun threading topic-lere daha chox yer ayirmisham. Bezi neshrlerde thread-i light-weight process de adlandirirlar. Process ise bir ve ya bir neche thread-den ibaret ola biler. Umumi anlamda, meqsed process-i bir neche thead-e bolmekle, paralelizm(multiprocessing) yaratmaqdir. Eslinde, bu, paralelizm deyil, concurrency(threading, asynchronism) prinsipidir. Thread-ler eyni memory space share edirler(stack ve register istisna), process-ler ise ferqli. Thread-ler bir-birine nezeren independent-dir, process-lere nezeren, hemchinin, process-ler de ozlerine nezeren depended-dir. Threading-de onlar arasi muxtelif nov data-larin share olunmasina baxmayaraq, stack(yeni, vahid stack) ve register-i share ede bilmirler. Chunki, her thread-in ozunun stack ve register-i olur. Onlar data segment-i(red. hecmhinin global & static variable-lar da bu segment-e yerleshirler) share ede bilirler.

Bezi arxitektura forumlarinda copy stack, copy local var. ve s. kimi qeyd edilir. Qisa olaraq, thread-ler hemchinin Kernel ve User level-lere de bolunurler. Hemishe bu ince meqami qeyd edirem ki, OS, user level thread-leri tanimasa da, bu level thread-lerde context switching daha tez icra olunur. Chunki, kernel level thread-lere eks olaraq, context switching uchun hardware support-a ehtiyyac duymur.

GIL esas etibarile Cpython2.7-de Guido terefinden 2003-cu ilde add edilib. Buna baxmayaraq, GIL locking-i Python-nun ilk multithreaded interpretatoru(1997-ci il) dovrunden movcud idi. Ashagidaki code fragment ve comment ona mexsusdur:

static PyThread_type_lock interpreter_lock = 0; /* This is the GIL */ https://github.com/python/cpython/blob/e62a694fee53ba7fc16d6afbaa53b373c878f300/Python/ceval.c#L238

Ona gore de ilk olaraq, kohne Python(CPython - byte code interpreter-lerinden biri) versiyalarindan research etmeye bashladim. Pythread_type_lock - Unix sistemlerinde standart C lock mexanizmidir. Interpretator ishe dushen kimi yuxaridaki kod setri de =0 deyer ile initialize olur ve butun kod strukturuna siraet edir:

void PyEval_InitThreads(void) { interpreter_lock = PyThread_allocate_lock(); PyThread_acquire_lock(interpreter_lock); } https://github.com/python/cpython/blob/master/Python/ceval.c#L198

Python thread-ler esas: POSIX(pthreads) --> gcc .c -lpthreads ve Windows(winpthreads). Thread-ler(red. butun thread-lere shamil olunmur) emeliyyat sistemi terefinden full manage olunurlar ve callable-dirlar. Thread uzerinden emeliyyatlar aparmaq uchun class verilib ve men onun uzerinden aparidiqim test neticelerini hisse-hisse qeyd edecem:

import time import threading

class ThreadC(): def __init__(self, count): threading.Thread.__init__(self) self.count = count

def call(self): while self.count > 0: print("Go down") self.count -= 1 time.sleep(5)

return

ThreadC(4).call()

Thread-ler CPU(low priority) ve I/O(high priority) bound novleri haqqinda.

Hemchinin, CPU bound thread-ler I/O interrupt-lara tabe olmurlar. Icra zamaninin(execution time) rough measurement testing-i:

# CPU bound functions(sequential) import time

def foo(n): while n > 0: n -= 1 # print(n)

start = time.clock() foo(1000000) foo(1000000) end = time.clock() print('result: ', (end - start)) >>> >>> >>> result: 0.10781199999999999

# CPU bound functions(threaded) import time from threading import Thread

def foo(n): while n > 0: n -= 1 # print(n)

start = time.clock() th1 = Thread(target=foo, args=(1000000, )) th1.start() th2 = Thread(target=foo, args=(1000000, )) th2.start() th1.join() th2.join() end = time.clock() print('result: ', (end - start)) >>> >>> >>> result: 0.178193

Neticeleri muqayise edib, aydin izahat vermek uchun deterministic dynamic analyze etmishem:

# Sequential >>> >>> >>> python -m cProfile -s ncalls traceseq.py

result: 0.1616552 8 function calls in 0.162 seconds

Ordered by: call count

ncalls tottime percall cumtime percall filename:lineno(function) 2 0.000 0.000 0.000 0.000 {built-in method time.clock} 2 0.162 0.081 0.162 0.081 seq.py:3(foo) 1 0.000 0.000 0.162 0.162 {built-in method builtins.exec} 1 0.000 0.000 0.000 0.000 {built-in method builtins.print} 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} 1 0.000 0.000 0.162 0.162 seq.py:1(<module>)

# Threaded >>> >>> >>> python -m cProfile -s ncalls tracethread.py

result: 0.35377359999999997 3429 function calls (3374 primitive calls) in 0.362 seconds

Ordered by: call count

ncalls tottime percall cumtime percall filename:lineno(function) 260 0.000 0.000 0.000 0.000 {method 'rstrip' of 'str' objects} 229 0.000 0.000 0.000 0.000 {method 'join' of 'str' objects} 219 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance} 180 0.000 0.000 0.000 0.000 {method 'add' of 'set' objects} 163 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects} 145/139 0.000 0.000 0.000 0.000 {built-in method builtins.len} 130 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:222(_verbose_message) 126 0.000 0.000 0.000 0.000 enum.py:267(__call__) 126 0.000 0.000 0.000 0.000 enum.py:517(__new__) 125 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:59(<listcomp>) 125 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:57(_path_join) 64 0.000 0.000 0.000 0.000 {method 'startswith' of 'str' objects} 57 0.000 0.000 0.000 0.000 enum.py:803(__and__) 54 0.000 0.000 0.000 0.000 sre_parse.py:232(__next) 52 0.000 0.000 0.000 0.000 {built-in method builtins.hasattr} 47 0.000 0.000 0.000 0.000 sre_parse.py:253(get) 45 0.000 0.000 0.000 0.000 {method 'rpartition' of 'str' objects} 41 0.000 0.000 0.000 0.000 sre_parse.py:163(__getitem__) 36 0.000 0.000 0.000 0.000 {method 'upper' of 'str' objects} 36 0.000 0.000 0.000 0.000 sre_parse.py:248(match) 35 0.002 0.000 0.002 0.000 {built-in method nt.stat} 35 0.000 0.000 0.002 0.000 <frozen importlib._bootstrap_external>:75(_path_stat) 30 0.000 0.000 0.000 0.000 {built-in method builtins.getattr} 30 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1080(_path_importer_cache) 28 0.000 0.000 0.000 0.000 {method 'find' of 'bytearray' objects} 25 0.000 0.000 0.000 0.000 {built-in method builtins.min} 25 0.000 0.000 0.000 0.000 {built-in method _imp.acquire_lock} 25 0.000 0.000 0.000 0.000 {built-in method _imp.release_lock} 25 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:37(_relax_case) 25 0.000 0.000 0.002 0.000 <frozen importlib._bootstrap_external>:1233(find_spec) 24 0.000 0.000 0.000 0.000 tokenize.py:148(<listcomp>) 20 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects} 20 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:321(<genexpr>) 20 0.000 0.000 0.000 0.000 sre_parse.py:171(append) 20 0.000 0.000 0.000 0.000 tokenize.py:112(group) 19 0.000 0.000 0.000 0.000 {built-in method builtins.__build_class__} 18 0.000 0.000 0.000 0.000 sre_parse.py:159(__len__) 17/8 0.000 0.000 0.000 0.000 sre_parse.py:173(getwidth) 17 0.000 0.000 0.000 0.000 sre_parse.py:285(tell) 16 0.000 0.000 0.000 0.000 {built-in method _thread.allocate_lock} 16 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:997(_handle_fromlist) 15 0.000 0.000 0.000 0.000 {built-in method nt.fspath} 15 0.000 0.000 0.000 0.000 {built-in method _thread.get_ident} 15 0.000 0.000 0.000 0.000 {built-in method builtins.ord} 15 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:843(__enter__) 15 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:847(__exit__) 12 0.353 0.029 0.353 0.029 {method 'acquire' of '_thread.lock' objects} 11 0.000 0.000 0.000 0.000 {method 'format' of 'str' objects} 11/2 0.000 0.000 0.000 0.000 sre_compile.py:64(_compile) 11 0.000 0.000 0.000 0.000 sre_parse.py:111(__init__) 10 0.000 0.000 0.000 0.000 {built-in method nt.getcwd} 10 0.000 0.000 0.000 0.000 {built-in method from_bytes} 10 0.000 0.000 0.000 0.000 {method 'rsplit' of 'str' objects} 10 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:52(_r_long) 10 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:403(cached) 10 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:63(_path_split) 10 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:263(cache_from_source) 8 0.000 0.000 0.000 0.000 sre_parse.py:294(_class_escape) 7/1 0.001 0.000 0.362 0.362 {built-in method builtins.exec} 6 0.000 0.000 0.000 0.000 {method '__contains__' of 'frozenset' objects} 6 0.000 0.000 0.000 0.000 {method 'isidentifier' of 'str' objects} 6 0.000 0.000 0.000 0.000 __init__.py:420(<genexpr>) 6 0.000 0.000 0.000 0.000 __init__.py:422(<genexpr>) 6 0.000 0.000 0.000 0.000 sre_compile.py:223(_compile_charset) 6 0.000 0.000 0.000 0.000 enum.py:797(__or__) 6 0.000 0.000 0.000 0.000 sre_compile.py:250(_optimize_charset) 6 0.000 0.000 0.000 0.000 sre_parse.py:81(groups) 5 0.000 0.000 0.000 0.000 {method 'read' of '_io.FileIO' objects} 5 0.000 0.000 0.000 0.000 {method 'extend' of 'list' objects} 5 0.000 0.000 0.000 0.000 {method 'endswith' of 'str' objects} 5 0.000 0.000 0.000 0.000 {built-in method builtins.any} 5 0.000 0.000 0.000 0.000 {built-in method _imp.is_builtin} 5 0.000 0.000 0.000 0.000 {built-in method _imp.is_frozen} 5 0.000 0.000 0.000 0.000 {built-in method _imp._fix_co_filename} 5 0.001 0.000 0.001 0.000 {built-in method marshal.loads} 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:35(_new_module) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:58(__init__) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:78(acquire) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:103(release) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:143(__init__) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:147(__enter__) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:151(__exit__) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:176(cb) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:157(_get_module_lock) 5/1 0.000 0.000 0.006 0.006 <frozen importlib._bootstrap>:211(_call_with_frames_removed) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:307(__init__) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:311(__enter__) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:318(__exit__) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:369(__init__) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:416(parent) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:424(has_location) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:504(_init_module_attrs) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:564(module_from_spec) 5/1 0.000 0.000 0.007 0.007 <frozen importlib._bootstrap>:651(_load_unlocked) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:707(find_spec) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:780(find_spec) 5 0.000 0.000 0.003 0.001 <frozen importlib._bootstrap>:870(_find_spec) 5/1 0.000 0.000 0.008 0.008 <frozen importlib._bootstrap>:936(_find_and_load_unlocked) 5/1 0.000 0.000 0.008 0.008 <frozen importlib._bootstrap>:966(_find_and_load) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:85(_path_is_mode_type) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:94(_path_isfile) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:361(_get_cached) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:393(_check_name_wrapper) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:430(_validate_bytecode_header) 5 0.000 0.000 0.001 0.000 <frozen importlib._bootstrap_external>:485(_compile_bytecode) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:524(spec_from_file_location) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:669(create_module) 5/1 0.000 0.000 0.007 0.007 <frozen importlib._bootstrap_external>:672(exec_module) 5 0.000 0.000 0.002 0.000 <frozen importlib._bootstrap_external>:743(get_code) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:800(__init__) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:825(get_filename) 5 0.000 0.000 0.001 0.000 <frozen importlib._bootstrap_external>:830(get_data) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:840(path_stats) 5 0.000 0.000 0.003 0.001 <frozen importlib._bootstrap_external>:1117(_get_spec) 5 0.000 0.000 0.003 0.001 <frozen importlib._bootstrap_external>:1149(find_spec) 5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1228(_get_spec) 5 0.000 0.000 0.000 0.000 sre_compile.py:388(_simple) 5 0.000 0.000 0.000 0.000 sre_parse.py:167(__setitem__) 5/2 0.000 0.000 0.000 0.000 sre_parse.py:470(_parse) 4 0.000 0.000 0.000 0.000 {method 'release' of '_thread.lock' objects} 4 0.000 0.000 0.000 0.000 sre_compile.py:539(isstring) 4/2 0.000 0.000 0.000 0.000 sre_parse.py:407(_parse_sub) 4 0.000 0.000 0.000 0.000 threading.py:506(is_set) 4 0.000 0.000 0.000 0.000 threading.py:1230(current_thread) 3 0.000 0.000 0.000 0.000 {method '__enter__' of '_thread.lock' objects} 3 0.000 0.000 0.000 0.000 {method '__exit__' of '_thread.lock' objects} 3 0.000 0.000 0.000 0.000 {method 'translate' of 'bytearray' objects} 3 0.000 0.000 0.000 0.000 {method 'items' of 'dict' objects} 3 0.000 0.000 0.000 0.000 _weakrefset.py:81(add) 3 0.000 0.000 0.000 0.000 sre_compile.py:378(<listcomp>) 3 0.000 0.000 0.000 0.000 sre_compile.py:376(_mk_bitmap) 3 0.000 0.000 0.000 0.000 threading.py:215(__init__) 3 0.000 0.000 0.000 0.000 threading.py:239(__enter__) 3 0.000 0.000 0.000 0.000 threading.py:242(__exit__) 3 0.000 0.000 0.000 0.000 threading.py:254(_is_owned) 3 0.000 0.000 0.000 0.000 threading.py:498(__init__) 3 0.000 0.000 0.000 0.000 threading.py:757(__init__) 3 0.000 0.000 0.000 0.000 tokenize.py:137(_all_string_prefixes) 2 0.000 0.000 0.000 0.000 {method 'append' of 'collections.deque' objects} 2 0.000 0.000 0.000 0.000 {built-in method _sre.compile} 2 0.000 0.000 0.000 0.000 {method 'locked' of '_thread.lock' objects} 2 0.000 0.000 0.000 0.000 {built-in method _thread.start_new_thread} 2 0.000 0.000 0.000 0.000 {method 'setter' of 'property' objects} 2 0.000 0.000 0.000 0.000 {method 'replace' of 'str' objects} 2 0.000 0.000 0.000 0.000 {built-in method builtins.max} 2 0.000 0.000 0.000 0.000 {built-in method time.clock} 2 0.000 0.000 0.001 0.001 re.py:231(compile) 2 0.000 0.000 0.001 0.001 re.py:286(_compile) 2 0.000 0.000 0.000 0.000 sre_compile.py:482(_compile_info) 2 0.000 0.000 0.001 0.000 sre_compile.py:542(_code) 2 0.000 0.000 0.001 0.000 sre_compile.py:557(compile) 2 0.000 0.000 0.000 0.000 sre_parse.py:76(__init__) 2 0.000 0.000 0.000 0.000 sre_parse.py:223(__init__) 2 0.000 0.000 0.000 0.000 sre_parse.py:828(fix_flags) 2 0.000 0.000 0.000 0.000 sre_parse.py:844(parse) 2 0.000 0.000 0.000 0.000 threading.py:248(_release_save) 2 0.000 0.000 0.000 0.000 threading.py:251(_acquire_restore) 2 0.000 0.000 0.018 0.009 threading.py:263(wait) 2 0.000 0.000 0.018 0.009 threading.py:533(wait) 2 0.000 0.000 0.000 0.000 threading.py:727(_newname) 2 0.000 0.000 0.018 0.009 threading.py:828(start) 2 0.000 0.000 0.000 0.000 threading.py:966(_stop) 2 0.000 0.000 0.336 0.168 threading.py:1024(join) 2 0.000 0.000 0.336 0.168 threading.py:1062(_wait_for_tstate_lock) 2 0.000 0.000 0.000 0.000 threading.py:1120(daemon) 2 0.000 0.000 0.000 0.000 tokenize.py:114(maybe) 1 0.000 0.000 0.000 0.000 {built-in method _thread._set_sentinel} 1 0.000 0.000 0.000 0.000 {method 'values' of 'dict' objects} 1 0.000 0.000 0.000 0.000 {method 'split' of 'str' objects} 1 0.000 0.000 0.000 0.000 {built-in method builtins.globals} 1 0.001 0.001 0.001 0.001 {built-in method builtins.print} 1 0.000 0.000 0.000 0.000 {built-in method builtins.repr} 1 0.000 0.000 0.000 0.000 {built-in method sys._getframe} 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} 1 0.000 0.000 0.000 0.000 _weakrefset.py:36(__init__) 1 0.000 0.000 0.001 0.001 __init__.py:357(namedtuple) 1 0.000 0.000 0.000 0.000 sre_compile.py:414(_get_literal_prefix) 1 0.000 0.000 0.000 0.000 sre_compile.py:441(_get_charset_prefix) 1 0.000 0.000 0.000 0.000 sre_parse.py:84(opengroup) 1 0.000 0.000 0.000 0.000 sre_parse.py:96(closegroup) 1 0.000 0.000 0.362 0.362 thread.py:1(<module>) 1 0.000 0.000 0.000 0.000 threading.py:87(_RLock) 1 0.000 0.000 0.000 0.000 threading.py:334(notify) 1 0.000 0.000 0.000 0.000 threading.py:357(notify_all) 1 0.000 0.000 0.000 0.000 threading.py:203(Condition) 1 0.000 0.000 0.000 0.000 threading.py:369(Semaphore) 1 0.000 0.000 0.000 0.000 threading.py:449(BoundedSemaphore) 1 0.000 0.000 0.000 0.000 threading.py:512(set) 1 0.000 0.000 0.000 0.000 threading.py:487(Event) 1 0.000 0.000 0.000 0.000 threading.py:566(Barrier) 1 0.000 0.000 0.000 0.000 threading.py:720(BrokenBarrierError) 1 0.000 0.000 0.000 0.000 threading.py:890(_set_ident) 1 0.000 0.000 0.000 0.000 threading.py:893(_set_tstate_lock) 1 0.000 0.000 0.000 0.000 threading.py:738(Thread) 1 0.000 0.000 0.000 0.000 threading.py:1158(Timer) 1 0.000 0.000 0.000 0.000 threading.py:1190(__init__) 1 0.000 0.000 0.000 0.000 threading.py:1188(_MainThread) 1 0.000 0.000 0.000 0.000 threading.py:1207(_DummyThread) 1 0.000 0.000 0.006 0.006 threading.py:1(<module>) 1 0.000 0.000 0.000 0.000 traceback.py:227(FrameSummary) 1 0.000 0.000 0.000 0.000 traceback.py:316(StackSummary) 1 0.000 0.000 0.000 0.000 traceback.py:438(TracebackException) 1 0.000 0.000 0.005 0.005 traceback.py:1(<module>) 1 0.000 0.000 0.004 0.004 linecache.py:6(<module>) 1 0.000 0.000 0.000 0.000 tokenize.py:99(TokenInfo) 1 0.000 0.000 0.000 0.000 tokenize.py:113(any) 1 0.000 0.000 0.000 0.000 tokenize.py:217(TokenError) 1 0.000 0.000 0.000 0.000 tokenize.py:219(StopTokenizing) 1 0.000 0.000 0.000 0.000 tokenize.py:222(Untokenizer) 1 0.000 0.000 0.003 0.003 tokenize.py:21(<module>) 1 0.000 0.000 0.000 0.000 token.py:74(<dictcomp>) 1 0.000 0.000 0.000 0.000 token.py:1(<module>) 1 0.000 0.000 0.000 0.000 <string>:5(TokenInfo) 1 0.000 0.000 0.000 0.000 <string>:1(<module>)

Valgrind graph based visualization almaq uchun onun front-side tool-undan (KCacheGrind) ve pyprof2calltree istifade etdim:

tracethread.py --> tracethread.cprof: python -m cProfile -o tracethread.cprof tracethread.py

kcachegrind format converting: pyprof2calltree -k -i tracethread.cprof

Ashagidaki qrafda esasen umumi tesvir gosterilib: main module, threading branches, call operations.

linecache.py --> threading.py --> token.py --> tokenize.py --> traceback.py --> tracethread.py

cProfile ncalls(’rstrip method of str’) neticelerle muqayise etmek uchun:

thread.start() - call() olunduqda, nece creat edilirler ve background-da hansi prosesler bash verir?! C dilinde(thread.h) - POSIX(1c~1b -API) pthreads type ve function-lar:

#include <pthread.h> int pthread_create(pthread_t *thread, const pthread_attr_t *attr, void *(*example)(void *), void *arg);

Ugurlu emeliyyatlarda pthread_create() sifir -0 return edir, qeyri-sifir deyerinde, <errno.h> -errno signal generate olunur.

*thread - pthread_t tipinde yaradilan thread-in id-sinin saxlanacagi address-dir.

*attr - pthread_attr_t thread attribute-larina typless(tipsiz) pointerdir. -NULL deyerde thread default attribute deyerlerle yaradilir.

*example - vahid deyishen movqeyinde void* type arqument qebul eden ve void* pointer return eden threaded function-a pointer-dir.

*arg - thread argument-leri saxlayan void* typless(tipsiz) pointer-dir. Ekser hallarda *arg global ve yaxud dynamic deyishene point edir. Chagirilan funksiya arqument teleb etmirse, o pozisiyada -NULL istifade ede bilerik.

pthread_t — thread identificator(axin ID); pthread_mutex_t — mutex; pthread_mutexattr_t — mutex attribute object(atribut obyekt); pthread_cond_t — conditional variable(sherti deyishen); pthread_condattr_t — conditional variable object attribute(sherti deyishenin obyekt atributu); pthread_key_t — thread specific datas(axina uygun verilenler); pthread_once_t — dynamic initialization control context(dinamiki elan edilme idare konteksti); pthread_attr_t — thread attribute schedule(axin atributu cedveli);

C(<thread.h>, <pthread.h>) & Python(threading)

Mesele artiq Python-luqdan chixib, ancaq, burda tam olaraq standart C-den de sohbet getmir, chunki, muzakire edilen thread-ler(pthreads) C-nin standartina(istisna <thread.h> windows os.) daxil deyil. Funksiya pointer-inin bu "numune funksiya" uzerinden daha aydin shekilde izahini vere bilmek uchun C dilinde bezi implementasiyalarla kichik kod example-lar yazmisham:

1) #include <stdio.h> #include <stdlib.h>

void *start(void *x){ printf("Function pointer\n"); }

int main(void) { void*(*example)(void *x); example = start; (*example)(NULL); // (*example)((void*)2);

return EXIT_SUCCESS; }

2) #include <stdio.h> #include <stdlib.h>

typedef void *(*Example)(void*);

void exampleLauncher(Example example){ (*example)(NULL); }

void *start(void *x){ printf("Function pointer\n"); }

int main(void) { Example example = start; exampleLauncher(example);

return EXIT_SUCCESS; }

3) #include <stdio.h> #include <stdlib.h>

typedef void *(*Example)(void*);

void exampleLauncher(Example example){ int numb = 5; (*example)((void*)&numb); // meselen: 57, 5, 9, 187 ve s. }

void *start(void *x){ printf("Function pointer\n"); int *y = (int*)x; printf("%d\n", *y); }

int main(void) { Example example = start; exampleLauncher(example);

return EXIT_SUCCESS; }

4) #include <stdio.h> #include <stdlib.h>

void *start(void *x){ printf("%d", *(int*)x); }

int main(void) { void*(*example)(void *x); example = start; int x = 5; (*example)(&x);

return EXIT_SUCCESS; }

Function pointer-lerle bagli bir neche vacib meqamdan biri de odur ki, simple pointer-lerden ferqli olaraq, data-ya yox, code-a point edirler. func() --> call by value ve f() --> call by referance - funksiyalarini data passing-e gore yazmisham:

#include <stdio.h> #include <stdlib.h>

void func(int x){ printf("address in func: %d\n", &x);

//return x; }

int f(int *x){ printf("address in f: \t %d\n", *&x);

return *x; }

int main(int argc, char **argv) { int x = 10; printf("address in main: %d\n", &x);

void (*fPtr)(int) = &func; (*fPtr)(x); //printf("%d\n", (*fPtr)(x));

int (*ptr)(int*) = &f; //(*ptr)(&x); printf("x value: \t %d\n", (*ptr)(&x));

return 0; }

gcc -o main *.c main ---------------------------- address in main: -1664382564 address in func: -1664382612 address in f: -1664382564 x value: 10

Esas meqsed (void*) qabariq shekilde gostermek idi. Hansi ki, simple void(no type return) deyil. void*(*example)(void*); ve void *start(void *x) uzlashmalidirlar. Eks halda aydin shekilde xeberdarliq aliriq:

warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]

Prosedurlara daha genish hakim ola bilmek uchun C dilinde thread-lerin ishleme prinsiplerini gosteren ve <unistd> -sleep() lib.-inden savayi oz sleep --> TSleep() funksiyami da yazdiqim sade bir proqram qurdum:

#include <stdio.h> #include <stdlib.h> #include <unistd.h> #include <pthread.h> #include <time.h>

void TSleep(int);

void *ThreadM(void *var) { TSleep(1); printf("Printing M-Thread\n"); TSleep(1); printf("var = %d\n", *((int *)var)); TSleep(1); }

void *ThreadS(void *val){ int *n = (int *)val; sleep(1); printf("\nPrinting S-Thread\n"); sleep(1); printf("val = %d\n", *n); sleep(1); }

int main(int argc, char **argv) { pthread_t thread_id; int sVal = 785; printf("Main: Before\n"); pthread_create(&thread_id, NULL, ThreadM, (void *)&sVal); pthread_join(thread_id, NULL); pthread_create(&thread_id, NULL, ThreadS, (void *)&sVal); pthread_join(thread_id, NULL); printf("Main: After\n"); exit(0); }

void TSleep(int sec){ int mSec = sec * 330000; // 1sec. => 1.000.000 µs; 1sec.* 1/3 clock_t tStart = clock(); printf("start: %ld\n", tStart); while(clock() < mSec + tStart); }

gcc threads.c -lpthread ./a.out ----------------------- Main: Before start: 641 Printing M-Thread start: 330675 var = 785 start: 660699

Printing S-Thread val = 785 Main: After

Python threading architecture resenziya(review)

1) Ilk olaraq, Python ozunde interpretatorun her hansi bir state-ini saxlayan kichik hecmli data structure(<100bytes) yaradir(creating process). 2) Burdan location alan yeni POSIX-thread launch(calling process) edir. 3) pthread ise PyEval_CallObject-i invoke edir. 4) Bu stage-de C function invoke olunur(Python callable):

/* Interpreter main loop */

PyObject * PyEval_EvalFrame(PyFrameObject *f) { /* This is for backward compatibility with extension modules that used this API; core interpreter code should call PyEval_EvalFrameEx() */ PyThreadState *tstate = _PyThreadState_GET(); return _PyEval_EvalFrame(tstate, f, 0); } https://github.com/python/cpython/blob/master/Python/ceval.c#L728

PyObject * PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) { PyThreadState *tstate = _PyThreadState_GET(); return _PyEval_EvalFrame(tstate, f, throwflag); } https://github.com/python/cpython/blob/master/Python/ceval.c#L738

Python C API <--> PyEval_CallObject-dir: PyEval_CallFunction(obj, "abc", a, b, c);- public API-in partition-i deyil. Limitlidir.

Chunki, PyAPI_FUNC-la bashlayan ve _Py ile bashlamayan function-lar hamisi public API-dir - https://mail.python.org/pipermail/python-dev/2011-February/107973.html

PyEval_CallFunction-la bagli official documentation-da hech ne tapa bilmedim - https://docs.python.org/3/search.html?q=PyEval_CallFunction

Ve, PyEval_CallObject(obj, Py_BuildValue("abc", a, b, c)); -in ekvivalentidir - https://docs.python.org/3/search.html?q=PyEval_CallObject

PyEval_* function-lara daxildir --> interpreter loop-un deyerlendirilmesi uchun "raw internal call"-lar.

PyObject_* functionlara ise --> interpretator-un elave "state integrity check"i, arqument validasiya, stack protection daxildir.

Bir chox API-lar undocumented olsalar bele, header file-larla mueyyen edilirler("#macro_definition").

PyEval_CallObject "#macro_definition"-dir --> PyEval_CallObjectWithKeywords: #define PyEval_CallObject(func, arg) \ PyEval_CallObjectWithKeywords(func, arg, (PyObject *)NULL)

Research elediyim qeder muxtelif forumlarda da qarshilashdiqim unofficial menbeler idi, hansi ki, bezilerinde, hetta, argument value position-da "str literal"-i circular brackets-le ishletmek lazim oldugu kimi qeydler edilirdi "(str literal)".

Bir qeder yuxari hissede 1)-ci bendde qeyd etdiyim kimi her thread-in spesifik interpretator data structur-i olur(Pythread state):

I - Python kod terefi uchun cari stack frame-i.

II - Cari rekursiya depth-i.

III - Thread ID.

IV - Her thread(per-thread) uchun exception info.

V - Tracing/profiling/debugging..hooks

Daha genish izah versek, thread butun informasiyani TCB(Thread Control Block)-da saxlayir.

TCB bloku: I - TID(Thread ID). II - Thread stack pointer --> Stack-in ozunde ise thread-in lokal deyishenleri yerleshirler. III - Thread state. IV - Register(Counter). Thread terefinden cari anda icra edilen instruction-un adresi yerleshir. V - Register set. VI - Parent process pointer --> PCB(Process Control Block) ise thread-in icra olundugu prosesin blokudur.

PCB bloku: I - PID(Process ID). II - Process state. III - Register(Counter). IV - Register set. V - Memory limits. VI - Achilan fayllarin cedveli.

TCB ve PCB <--> Process memory(stack, data, text) ile qarshiliqli elaqede fealiyyet gosterirler.

Interpretatorun GIL-i oldugu kimi GIV-si(Global Interpreter Variable) de var. Cari anda run olunan thread-in ThreadState(PyThreadState) strukturuna pointer-dir. Interpretator emeliyyatlar hansi thread-lerle ishlediklerini bilmeleri uchun implicit olaraq, GIV-den asili olurlar:

PyThreadStateStructure: typedef struct _ts { struct _ts *next; PyInterpreterState *interp; struct _frame *frame; int recursion_depth; int tracing; int use_tracing; Py_tracefunc c_profilefunc; Py_tracefunc c_tracefunc; PyObject *c_profileobj; PyObject *c_traceobj; PyObject *curexc_type; PyObject *curexc_value; PyObject *curexc_traceback; PyObject *exc_type; PyObject *exc_value; PyObject *exc_traceback; PyObject *dict; int tick_counter; int gilstate_counter; PyObject *async_exc; long thread_id; } PyThreadState; // --> PyThreadStateStructure

Thread execution: /* Python/pystate.c */ ... PyThreadState *_PyThreadState_Current = NULL;

Old GIL dovrunde interpretator CPU bound thread-leri check elemek uchun her 100 interpretator anindan bir check edirdi. Yeni, checking her 100 tickden bir olurdu. Bu, independed global counter sayilir(thread scheduling-den asili deyil). Bu tick-ler interpretator instruction-lara hardasa yaxindir. Hemchinin, tick-ler time-based deyil(di). Long operation-lar diger emeliyyatlar blok ede biler(di)ler,- uninterruptable olduglari uchun:

values = xrange(n) * - Python 3-de rename edilib range(n) xrange = range values = xrange(100000000) print(-1 in values) --> False

>>> def tick_exe(n_values): ... xrange = range * - it's not must to do. Just for visualize. ... values = xrange(n_values) ... start = time.clock() ... print(-1 in values) ... end = time.clock() ... return end - start ... >>> tick_exe(1000) False 0.00023360000000138825 >>> tick_exe(100000) False 0.0012785000000121727 >>> tick_exe(1000000) False 0.0012816000000839267 >>> tick_exe(1000000000000) False 0.00022339999998166604 >>> tick_exe(1000000000000000000000000000000) False 0.0012336000000914282 >>>

Thread-ler arasi interaction-i, hemchinin, thread-ler ve GIL arasindaki interaction-lari izlemek, bu eventleri basha dushmek uchun onlari log-layaraq, muqayiseler apardim:

# CPU bound functions logging: import time import logging import threading

def foo(n): logging.info("%s: starting", n)

try: while n > 0: n -= 1 logging.info("%s: starting", n) finally: logging.debug("End")

if __name__ == '__main__': start = time.clock() format = "%(asctime)s: %(message)s" logging.basicConfig(level=logging.DEBUG, format='(%(threadName)-9s) %(message)s',)

logging.info("Main : before creating thread") th1 = threading.Thread(target=foo, args=(10, )) logging.info("Main : before running thread(acquire)") th1.start() logging.info("Main : wait for the thread to finish(release)") logging.info("Ext : before creating thread") th2 = threading.Thread(target=foo, args=(10, )) logging.info("Ext : before running thread(acquire)") th2.start() logging.info("Ext : wait for the thread to finish(release)") th1.join() logging.info("Main : thread done") th2.join() logging.info("Ext : thread done") end = time.clock() print('result: ', (end - start)) >>> >>> >>> (MainThread) Main : before creating thread (MainThread) Main : before running thread(acquire) (Thread-1 ) 10: starting (Thread-1 ) 9: starting (Thread-1 ) 8: starting (Thread-1 ) 7: starting (Thread-1 ) 6: starting (Thread-1 ) 5: starting (Thread-1 ) 4: starting (Thread-1 ) 3: starting (Thread-1 ) 2: starting (Thread-1 ) 1: starting (Thread-1 ) 0: starting (Thread-1 ) End (MainThread) Main : wait for the thread to finish(release) (MainThread) Ext : before creating thread (MainThread) Ext : before running thread(acquire) (Thread-2 ) 10: starting (Thread-2 ) 9: starting (Thread-2 ) 8: starting (Thread-2 ) 7: starting (Thread-2 ) 6: starting (Thread-2 ) 5: starting (Thread-2 ) 4: starting (Thread-2 ) 3: starting (Thread-2 ) 2: starting (Thread-2 ) 1: starting (Thread-2 ) 0: starting (Thread-2 ) End (MainThread) Ext : wait for the thread to finish(release) (MainThread) Main : thread done (MainThread) Ext : thread done result: 0.0027679999999999996

p.s.: bele crash hallar da olur. OS yene de mudaxil edir ve execution queue scenario-nu ozu teyin edir. >>> >>> >>> (MainThread) Main : before creating thread (MainThread) Main : before running thread(acquire) (Thread-1 ) 10: starting (Thread-1 ) 9: starting (MainThread) Main : wait for the thread to finish(release) (MainThread) Ext : before creating thread (MainThread) Ext : before running thread(acquire) (Thread-1 ) 8: starting (Thread-1 ) 7: starting (Thread-1 ) 6: starting (Thread-1 ) 5: starting (Thread-1 ) 4: starting (Thread-1 ) 3: starting (Thread-1 ) 2: starting (Thread-1 ) 1: starting (Thread-1 ) 0: starting (Thread-1 ) End (Thread-2 ) 10: starting (Thread-2 ) 9: starting (Thread-2 ) 8: starting (MainThread) Ext : wait for the thread to finish(release) (Thread-2 ) 7: starting (MainThread) Main : thread done (Thread-2 ) 6: starting (Thread-2 ) 5: starting (Thread-2 ) 4: starting (Thread-2 ) 3: starting (Thread-2 ) 2: starting (Thread-2 ) 1: starting (Thread-2 ) 0: starting (Thread-2 ) End (MainThread) Ext : thread done

result: 0.004022000000000005

Bu hisseye qeder ancaq time.clock() istifade etmishem. Qeyd edim ki, clock() Unix platformalarinda cari CPU sec. time-i return edir. Testleri Linux sistemi uzerinde aparmisham. Butun platformalarda eyni behaviour-a sahib time.time() da istifade etmek olar. Automation script numune de vere bilerik:

import sys if sys.platform == 'win32': timer = time.clock else: timer = time.time

Zeruri deqiqliye ehtiyyac yaranarsa, timeit()(red. by default gc.desable tetbiq edir) ishlede bilerik.

Thread-ler keyboard interrupting-le kill olmur(du)lar. Eger, signal handle olundusa, interpretator tick's check-i 100 tick-den bir deyil, her tick-den bir edir(di). Signal handler ancaq main thread-de run oldugu uchun GIL her tick-den sonra acquire/release edir(di)(scheduled olunana qeder). Amma, maraqli meqam var. Ki, Python-un thread scheduler-i yox idi. Bu ishleri host OS-e buraxirdi(Windows, Linux, etc.). Thread-lerle bagli delaying prosesler, qeribe qanunauygunsuzluqlar ve s. prosedurlar bununla bagli idi ki, Python interpretator thread-leri control etmir, eksine, thread-ler arasi switching-i icra edirdi:

Thread switching in new GIL: pseudo /* Python/ceval.c */ ... static volatile int gil_drop_request = 0;

Bashqa meqam frozen signal-larla baglidir

Signal handling-in bir qoludur. Hansi ki, interpreter her tick-den sonra switch etmek isteyir ve frozen signals bash verir. Signal overhead de yarana bilir - Interpreter ticks, locks, waiting prosesler(semaphore) ve s. gore. Bununla bagli sonda yene qeydler aparmisham. Bildiyimiz kimi Python-da kod emali 2 esas etapa bash verir. Code compiling-den sonra byte-code erseye gelir. Buna gore de main project directory-de .pyc ve ya __pycache__ folderi auto creat olur. Bu kod da Python interpreter terefinden loop-da oxunulur ve butun instruction-lar step-by-step icra olunur. Bu muddet erzinde o cari thread-den sorushmadan, GIL drop ede bilir.. Bu uzden de diger thread-leri de run ede biler:

for (;;) { if (--ticker < 0) { ticker = check_interval;

/* Give another thread a chance */ PyThread_release_lock(interpreter_lock);

/* Other threads may run now */ PyThread_acquire_lock(interpreter_lock, 1); }

bytecode = *next_instr++; switch (bytecode) { /* execute the next instruction ... */ } }

for loop() blokunda olan procedure ve function-larin source code-lari:

void PyThread_release_lock(PyThread_type_lock aLock) { dprintf(("%lu: PyThread_release_lock(%p) called\n", PyThread_get_thread_ident(),aLock));

if (!(aLock && LeaveNonRecursiveMutex((PNRMUTEX) aLock))) dprintf(("%lu: Could not PyThread_release_lock(%p) error: %ld\n", PyThread_get_thread_ident(), aLock, GetLastError())); } https://github.com/python/cpython/blob/master/Python/thread_nt.h#L340

int PyThread_acquire_lock(PyThread_type_lock aLock, int waitflag) { return PyThread_acquire_lock_timed(aLock, waitflag ? -1 : 0, 0); } https://github.com/python/cpython/blob/master/Python/thread_nt.h#L334

Yeni GIL thread switching-in opcode counting(_Py_Ticker ile 100 interpreter ani) ile yerine yetirmek evezine fixed interval istifade edir:

"""Old version""": /* Python/ceval.c */ ... if (--_Py_Ticker < 0) { ... _Py_Ticker = _Py_CheckInterval; ... if (things_to_do) { if (Py_MakePendingCalls() < 0) { ... } } if (interpreter_lock) { /* Give another thread a chance */ ... PyThread_release_lock(interpreter_lock); /* Other threads may run now */ PyThread_acquire_lock(interpreter_lock, 1); ... }

""" Rewriting - New GIL concept by Antoine Pitrou: """ import sys sys.setcheckinterval(n) # n - fixed interval value

Thread switching latency prosesinin qarshisini almaq uchun kimi 2 esas mexanizmden istifade edir:

1) Forced thread switching - Latency evvelde bildirdiyimiz kimi task scheduling-in OS terefinden planlanilmasina gore bash verirdi. GIL-i acquire eden thread switching interval bitende, yeniden bu thread GIL-i catch ede bilmir. 2) Priority requests - Bir thread GIL-e sahib olanda, diger thread GIL-i catch etmek istedikde, onun request-i --> priority request kimi deyerlendirilir ve task scheduling-de ilk icra olunacaqlar siyahisina alinir(movqeyine qoyulur). Hemchinin, cari thread chox tez zamanda GIL-i release edir: (eval)

Thread switching in new GIL: pseudo /* Python/ceval.c */ ... static volatile int gil_drop_request = 0;

Bes, multicore GIL contention(instruction) nece bash verir ?!

Multicores-da CPU-bound thread-ler eyni anda scheduled edilir(diqqet, ferqli core-larda) ve ardi ile GIL battle bash verir. Old GIL battle-in tesviri:

Old GIL battle trace:

t2 100 5392 ENTRY t2 100 5392 ACQUIRE t2 100 5393 RELEASE t1 100 5393 ACQUIRE t2 100 5393 ENTRY t2 27 5393 BUSY t1 100 5394 RELEASE t1 100 5394 ENTRY t1 100 5394 ACQUIRE t2 74 5394 RETRY t1 100 5395 RELEASE t1 100 5395 ENTRY t1 100 5395 ACQUIRE t2 83 5395 RETRY t1 100 5396 RELEASE t1 100 5396 ENTRY t1 100 5396 ACQUIRE t2 80 5396 RETRY t1 100 5397 RELEASE t1 100 5397 ENTRY t1 100 5397 ACQUIRE t2 79 5397 RETRY

Belece, thread 2-nin wait() time-i 100-lerle faild-le davam ede biler. Sonlanmasa(extern olaraq), successful olana kimi.

GIL battle ise schedulers conflict-e sebeb olur. Python single threading-e ustunluk verir. Ve, thread scheduling-le bagli her nesneni host OS-e hevale edir. Host OS ise daha chox avantaj elde etmek uchun mumkun qeder process/thread-leri daha chox core-dan istifade etmeye chalishir.

MultiCore GIL contention I/O bound thread-lerin response time-ini uzadir:

I/O bound trace: t2 100 161241 ACQUIRE t2 100 161242 RELEASE t2 100 161242 ENTRY t2 100 161242 ACQUIRE t2 100 161243 RELEASE t2 100 161243 ENTRY t2 100 161243 ACQUIRE t1 45 161243 ENTRY t1 38 161243 BUSY t2 100 161244 RELEASE t2 100 161244 ENTRY t2 100 161244 ACQUIRE t1 68 161244 RETRY t2 100 161245 RELEASE t2 100 161245 ENTRY t2 100 161245 ACQUIRE t1 77 161245 RETRY ... t1 100 161404 ACQUIRE t1 97 161404 RELEASE

Baxmayaraq ki, CPU bound thread-ler low-priority-di, I/O bound thread-lerin execute olunmasini bloklaya bilirler. Hemchinin, I/O bound thread-ler GIL acquire etmek uchun fast wake mode-a malik deyiller ki, CPU bound thread GIL-i reacquire edende, bunu icra etsinler. Vacib meqam odur ki, new GIL time-based-dir(seconds). C extension ve calculative emeliyyatlara wait time choxdur.

Yeni GIL illustration-da I thread ishleyende, II thread wait mode-da olur. Ve, I thread-in release etmeyini gozleyir. Bu prosesde I thread ya input/output edir, ya da sleep mode-a kechir. New GIL battle-in vizual tesviri. 3-cu qrafik colloborative multitasking-i ehate edir:

Yeni GIL thread-e 5~15ms run olmaga icaze verir,- I/O bound thread de daxil olmaqla. Lakin, CPU bond thread yene de prioriteti qoruyur,- I/O bound-lari bloklaya bilir. Bu ise, yene de trashing, context switching-de problemler yarada biler.

Bu hisse threading-de bezi yeni yanashmalari kichik proqramlar uzerinden test neticelerinden behs edir: Daemon, ThreadPoolExecutor, Lock, RLock ve s.

Daemon - background thread-dir. Github-da cpython/threading kodlarindan gorunur ki, execution time bitende, threading_shutdown() butun chagirilan thread-leri yoxlayaraq, hansinda ki, daemon flag yoxdur, thread uchun join() chagirir. daemon flag = True olanda, thread-in sonlanmasi gozlenilmir,- exit() olunur. Bu numunede time.sleep(2) istifade edilib. Eksine, daemon flag-siz halda proqram chixishi gozleyir. Bele ki, thread ozu sleep mode-da olur:

import logging import threading import time

def thread_function(name): logging.info("Thread %s: starting", name) time.sleep(2) logging.info("Thread %s: finishing", name)

if __name__ == "__main__": format = "%(asctime)s: %(message)s" logging.basicConfig(format=format, level=logging.INFO, datefmt="%H:%M:%S")

logging.info("Main : before creating thread") x = threading.Thread(target=thread_function, args=(1,)) logging.info("Main : before running thread") x.start() logging.info("Main : wait for the thread to finish") # x.join() logging.info("Main : all done") >>> >>> >>> 12:36:03: Main : before creating thread 12:36:03: Main : before running thread 12:36:03: Thread 1: starting 12:36:03: Main : wait for the thread to finish 12:36:03: Main : all done 12:36:05: Thread 1: finishing

Ashagidaki ssenariye uygun code fragment-de ise chixishda proqram exit() olur ve buna gore daemon kill() olur deye, thread finish olmur:

import logging import threading import time

def thread_function(name): logging.info("Thread %s: starting", name) time.sleep(2) logging.info("Thread %s: finishing", name)

if __name__ == "__main__": format = "%(asctime)s: %(message)s" logging.basicConfig(format=format, level=logging.INFO, datefmt="%H:%M:%S")

logging.info("Main : before creating thread") x = threading.Thread(target=thread_function, args=(1,), daemon=True) logging.info("Main : before running thread") x.start() logging.info("Main : wait for the thread to finish") # x.join() logging.info("Main : all done") >>> >>> >>> 12:51:05: Main : before creating thread 12:51:05: Main : before running thread 12:51:05: Thread 1: starting 12:51:05: Main : wait for the thread to finish 12:51:05: Main : all done

Eger, ardicilligin qorunmasini isteyirikse, thread finish edilib, proqramin exit() olmasi uchun hem daemon=True, hem de join() ishlenmelidir. Ilk, thread finish() olur, sonra proqram exit() olur(continuous queue):

import logging import threading import time

def thread_function(name): logging.info("Thread %s: starting", name) time.sleep(2) logging.info("Thread %s: finishing", name)

if __name__ == "__main__": format = "%(asctime)s: %(message)s" logging.basicConfig(format=format, level=logging.INFO, datefmt="%H:%M:%S")

logging.info("Main : before creating thread") x = threading.Thread(target=thread_function, args=(1,), daemon=True) logging.info("Main : before running thread") x.start() logging.info("Main : wait for the thread to finish") x.join() logging.info("Main : all done") >>> >>> >>> 12:54:32: Main : before creating thread 12:54:32: Main : before running thread 12:54:32: Thread 1: starting 12:54:32: Main : wait for the thread to finish 12:54:34: Thread 1: finishing 12:54:34: Main : all done

Multithreading application-larda ise start() ve join() ardicilliglari chox ferqlenirler. Bunu OS ozu xaotik mentiqle nizamlayir ve bu uzden predict etmek chox chetindir,- ferqli neticeler emal olunur. Bes, bu prosedurlar nece hasil olunurlar?!

M2M, M2O, O2O - modellerinin ishleme prinsipleri ve vizualizasiyalari

Multithread app.-lar multithread modeller(many-to-many(M2M), many-to-one(M2O), one-to-one(O2O)) sayesinde erseye gelir:

M2M - model relation-da example gotursek, 4 user level threads iki-iki formada subprocess-ler kimi ayri process-ler daxilinde cemleshirler ve unit transfer edilirler kernel level-e, 4 kernel level thread-e bolunurler, ardi ile CPU cores-a nezeren optimal paylanaraq, paralelizmi temin etmeye chalishirlar.

M2O - yox eger, user thread lib.-ler OS terefinden support edilmirse, bu halda M2O(concurency principle) model tetbiq edilir. Bir thread-e sys. call blocking tetbiq edilse, aid oldugu process de bloklanir. Kernel level-e process daxilinden ancaq bir thread-in access-i olur.

O2O(>M2O) - esas ferq bir thread-e sys. call blocking zamani diger thread Kernel-e access ede bilir.

import logging import threading import time

def thread_function(name): logging.info("Thread %s: starting", name) time.sleep(2) logging.info("Thread %s: finishing", name)

if __name__ == "__main__": format = "%(asctime)s: %(message)s" logging.basicConfig(format=format, level=logging.INFO, datefmt="%H:%M:%S")

threads = list() for index in range(3): logging.info("Main : create and start thread %d.", index) x = threading.Thread(target=thread_function, args=(index,)) threads.append(x) x.start()

for index, thread in enumerate(threads): logging.info("Main : before joining thread %d.", index) thread.join() logging.info("Main : thread %d done", index) >>> >>> >>> // 1-ci execution result: 13:02:20: Main : create and start thread 0. 13:02:20: Thread 0: starting 13:02:20: Main : create and start thread 1. 13:02:20: Thread 1: starting 13:02:20: Main : create and start thread 2. 13:02:20: Thread 2: starting 13:02:20: Main : before joining thread 0. 13:02:22: Thread 0: finishing 13:02:22: Main : thread 0 done 13:02:22: Main : before joining thread 1. 13:02:22: Thread 1: finishing 13:02:22: Main : thread 1 done 13:02:22: Main : before joining thread 2. 13:02:22: Thread 2: finishing 13:02:22: Main : thread 2 done

>>> // 2-ci execution result: 13:05:59: Main : create and start thread 0. 13:05:59: Thread 0: starting 13:05:59: Main : create and start thread 1. 13:05:59: Thread 1: starting 13:05:59: Main : create and start thread 2. 13:05:59: Thread 2: starting 13:05:59: Main : before joining thread 0. 13:06:01: Thread 2: finishing 13:06:01: Thread 1: finishing 13:06:01: Thread 0: finishing 13:06:01: Main : thread 0 done 13:06:01: Main : before joining thread 1. 13:06:01: Main : thread 1 done 13:06:01: Main : before joining thread 2. 13:06:01: Main : thread 2 done

ThreadPoolExecutor() - continous queue qurulumasi uchun FIFO prinsipi ile ishleyen diger usuldur. Lazim olacaq thread-lerin sayi verilir. Meselen: workers=4. Bu thread-ler Pool-da execute edilir,- adindan gorunduyu kimi. Sonra, .map() ile uzerlerinden iterate() edilir. Hemchinin, with block-unun sonundaki executor join() method-unu explicitly manual gostermesek bele, icra edir. Amma, bezi hallarda: Xususile, threaded function istifade edirikse ve function parameter qebul elemirse, lakin, .map() istifade edirikse, muxtelif OS-lerde exception generate ola biler:

import logging import threading import concurrent.futures import time

def thread_function(name): logging.info("Thread %s: starting", name) time.sleep(2) logging.info("Thread %s: finishing", name)

# [rest of code]

if __name__ == "__main__": format = "%(asctime)s: %(message)s" logging.basicConfig(format=format, level=logging.INFO, datefmt="%H:%M:%S")

with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor: executor.map(thread_function, range(3)) >>> >>> >>> 13:27:56: Thread 0: starting 13:27:56: Thread 1: starting 13:27:56: Thread 2: starting 13:27:58: Thread 0: finishing 13:27:58: Thread 1: finishing 13:27:58: Thread 2: finishing

Ola bilecek error-un qarshisini almaq uchun .map() deyil, .submit() istifade edirem. Hem de burda ssenari ferqlidir. Database-deki value-nu method vasitesile update edirem. update() her thread-e gore icra olunur deye, her thread oz local value-sunu update etmish olur ve burda race condition ve ya oxshar hallar(undefined behavior) meydana chixir. Buna instance method deyilir. Ve, yuxarida yazdigim kimi bu variable-larin thread oriented oldugunu goruruk. Race condition, threading proqramlarinda meydana chixan en genish yayilmish problemlerden biridir. Hardasa, xeta da adlandirmaq olar. Termin elektronikadan gelmedir. Proqram teminatinda ve ya elektronik muhitde(sxemde) icra ardicilliginin pozulmasi, qarishiqligin mushahide olunmasidir. Bele xetalar, hansi ki, oz davranishini deyishir "heisenbug" adlanir. Dahi fizik, nobel mukafati laureati, kvant mexanikasinin qurucularindan olan Heisenberg-in adi ile baglidir. Buna oxshar bug type da var, lakin, xaotik davranish sergileyir,- "mandelbug" adlanir. Boyuk amerika-fransa riyaziyyatchisi, fizik Mandelbrot-un sherefine adlandirilib. O fraktal hendesenin banisidir. Riyaziyyatda fraktal dedikde, Evklid muhitinde topoloji hendesi fiqurlardan olchulerine ve diger baglantilarina gore ferqlenen noqteler nezerde tutulur. Nese, movzudan konkret uzaqlashdim)))))) Qisaca, database-deki value-nu update etmek uchun bele yanashma duzgun deyil:

import logging import threading import concurrent.futures import time

class FakeDatabase: def __init__(self): self.value = 0

def update(self, name): logging.info("Thread %s: starting update", name) local_copy = self.value local_copy += 1 time.sleep(0.1) self.value = local_copy logging.info("Thread %s: finishing update", name)

if __name__ == "__main__": format = "%(asctime)s: %(message)s" logging.basicConfig(format=format, level=logging.INFO, datefmt="%H:%M:%S")

database = FakeDatabase() logging.info("Testing update. Starting value is %d.", database.value) with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor: for index in range(2): executor.submit(database.update, index) logging.info("Testing update. Ending value is %d.", database.value) >>> >>> >>> 14:31:41: Testing update. Starting value is 0. 14:31:42: Thread 0: starting update 14:31:42: Thread 1: starting update 14:31:42: Thread 0: finishing update 14:31:42: Thread 1: finishing update 14:31:42: Testing update. Ending value is 1.

Race condition-un qarshisini almaq uchun: Lock (diger proqramlashdirma dillerinde mutex ve s.), RLock(ext), Server-Client, Server-Client(Lock), Server-Client(Queue) istifade edile biler. Lock() - context manager rolunu ifa edir. Yeni ki, context switching operation-lari icra edir. Vahid zaman erzinde yalniz bir thread Lock()-u catch(acquire) ede biler. Diger thread Lock()-a sahib olmaq uchun gozlemelidir ki, Lock() release edilsin:

import logging import threading import concurrent.futures import time

class FakeDatabase: def __init__(self): self.value = 0 self._lock = threading.Lock()

def locked_update(self, name): logging.info("Thread %s: starting update", name) logging.debug("Thread %s about to lock", name) with self._lock: logging.debug("Thread %s has lock", name) local_copy = self.value local_copy += 1 time.sleep(0.1) self.value = local_copy logging.debug("Thread %s about to release lock", name) logging.debug("Thread %s after release", name) logging.info("Thread %s: finishing update", name)

if __name__ == "__main__": format = "%(asctime)s: %(message)s" logging.basicConfig(format=format, level=logging.INFO, datefmt="%H:%M:%S")

database = FakeDatabase() logging.info("Testing update. Starting value is %d.", database.value) with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor: for index in range(2): executor.submit(database.locked_update, index) # logging.getLogger().setLevel(logging.DEBUG) # full logging uchun logging.info("Testing update. Ending value is %d.", database.value) >>> >>> >>> 16:35:37: Testing update. Starting value is 0. 16:35:37: Thread 0: starting update 16:35:37: Thread 1: starting update 16:35:37: Thread 1 about to lock 16:35:37: Thread 0 about to release lock 16:35:37: Thread 0 after release 16:35:37: Thread 0: finishing update 16:35:37: Thread 1 has lock 16:35:37: Thread 1 about to release lock 16:35:37: Thread 1 after release 16:35:37: Thread 1: finishing update 16:35:37: Testing update. Ending value is 2.

RLock() - Lock()-un bir sira chatishmazliqlarini hell ede bilerem. Lock()-da ilk thread acquire() edilib, release() edile bilinmeyende(livelock), diger thread-ler Lock()-u catch ede bilmirler(deadlock). Bu da wait mode-da olan thread-ler arasinda aktiv ishtirakchi olmasalar bele, oz aralarinda queue behavior-larini tez-tez deyishmelerine, burdan da elave problemlere yol achilmasina sebeb olur. Esas ferq ondan ibaretdir ki, Lock() ferqli thread terefinden catch(bir thread terefinden bir defe) ve tamam ayri bir thread terefinden release oluna biler. RLock()-da ise bir thread terefinden bir neche defe catch ediler biler ve s.:

import logging import threading import concurrent.futures import time

class FakeDatabase: def __init__(self): self.value = 0 self._lock = threading.RLock() # attention

if __name__ == "__main__": format = "%(asctime)s: %(message)s" logging.basicConfig(format=format, level=logging.INFO, datefmt="%H:%M:%S")

database = FakeDatabase() logging.info("Testing update. Starting value is %d.", database.value) with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor: for index in range(2): executor.submit(database.locked_update, index) #logging.getLogger().setLevel(logging.DEBUG) logging.info("Testing update. Ending value is %d.", database.value) >>> >>> >>> 16:37:07: Testing update. Starting value is 0. 16:37:07: Thread 0: starting update 16:37:07: Thread 1: starting update 16:37:07: Thread 1 about to lock 16:37:07: Thread 1 has lock 16:37:07: Thread 0 about to lock 16:37:08: Thread 1 about to release lock 16:37:08: Thread 1 after release 16:37:08: Thread 1: finishing update 16:37:08: Thread 0 has lock 16:37:08: Thread 0 about to release lock 16:37:08: Thread 0 after release 16:37:08: Thread 0: finishing update 16:37:08: Testing update. Ending value is 2.

Server-Client(Lock) model dizayni. Mesele ondan ibaretdir ki, cari anda konveyer(SENTINEL obyekti) ancaq bir deyer saxlaya bilir. Ya Server, ya da Client:

import random import logging import threading import concurrent.futures

SENTINEL = object()

class Pipeline: """ Class to allow a single element pipeline between producer and consumer. """ def __init__(self): self.message = 0 self.producer_lock = threading.Lock() self.consumer_lock = threading.Lock() self.consumer_lock.acquire()

def get_message(self, name): self.consumer_lock.acquire() message = self.message self.producer_lock.release() return message

def set_message(self, message, name): self.producer_lock.acquire() self.message = message self.consumer_lock.release()

def producer(pipeline): """Pretend we're getting a message from the network.""" for index in range(10): message = random.randint(1, 101) logging.info("Producer got message: %s", message) pipeline.set_message(message, "Producer")

# Send a sentinel message to tell consumer we're done pipeline.set_message(SENTINEL, "Producer")

def consumer(pipeline): """Pretend we're saving a number in the database.""" message = 0 while message is not SENTINEL: message = pipeline.get_message("Consumer") if message is not SENTINEL: logging.info("Consumer storing message: %s", message)

if __name__ == "__main__": format = "%(asctime)s: %(message)s" logging.basicConfig(format=format, level=logging.INFO, datefmt="%H:%M:%S") # logging.getLogger().setLevel(logging.DEBUG)

pipeline = Pipeline() with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor: executor.submit(producer, pipeline) executor.submit(consumer, pipeline)

>>> >>> >>> 16:43:23: Producer got message: 67 16:43:23: Producer got message: 79 16:43:23: Consumer storing message: 67 16:43:23: Consumer storing message: 79 16:43:23: Producer got message: 22 16:43:23: Producer got message: 77 16:43:23: Consumer storing message: 22 16:43:23: Producer got message: 40 16:43:23: Consumer storing message: 77 16:43:23: Producer got message: 54 16:43:23: Consumer storing message: 40 16:43:23: Producer got message: 97 16:43:23: Consumer storing message: 54 16:43:23: Producer got message: 89 16:43:23: Consumer storing message: 97 16:43:23: Producer got message: 53 16:43:23: Consumer storing message: 89 16:43:23: Producer got message: 37 16:43:23: Consumer storing message: 53 16:43:23: Consumer storing message: 37

Server-Client(Queue) model dizayni. Yuxaridaki modelden ferqi odur ki, konveyer eyni anda 1-den chox deyer saxlaya bilir.

import concurrent.futures import logging import queue import random import threading import time

def producer(queue, event): """Pretend we're getting a number from the network.""" while not event.is_set(): message = random.randint(1, 101) logging.info("Producer got message: %s", message) queue.put(message)

logging.info("Producer received event. Exiting")

def consumer(queue, event): """Pretend we're saving a number in the database.""" while not event.is_set() or not queue.empty(): message = queue.get() logging.info( "Consumer storing message: %s (size=%d)", message, queue.qsize() )

logging.info("Consumer received event. Exiting")

if __name__ == "__main__": format = "%(asctime)s: %(message)s" logging.basicConfig(format=format, level=logging.INFO, datefmt="%H:%M:%S")

pipeline = queue.Queue(maxsize=10) event = threading.Event() with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor: executor.submit(producer, pipeline, event) executor.submit(consumer, pipeline, event)

time.sleep(0.1) logging.info("Main: about to set event") event.set()

>>> >>> >>> 16:50:44: Consumer storing message: 10 (size=9) 16:50:44: Consumer storing message: 21 (size=9) 16:50:44: Producer got message: 49 16:50:44: Producer got message: 43 16:50:44: Consumer storing message: 75 (size=9) 16:50:44: Consumer storing message: 100 (size=9) 16:50:44: Producer got message: 37 16:50:44: Main: about to set event 16:50:44: Producer got message: 83 16:50:44: Consumer storing message: 97 (size=9) 16:50:44: Consumer storing message: 82 (size=9) 16:50:44: Producer received event. Exiting 16:50:44: Consumer storing message: 16 (size=9) 16:50:44: Consumer storing message: 64 (size=8) 16:50:44: Consumer storing message: 52 (size=7) 16:50:44: Consumer storing message: 20 (size=6) 16:50:44: Consumer storing message: 82 (size=5) 16:50:44: Consumer storing message: 65 (size=4) 16:50:44: Consumer storing message: 49 (size=3) 16:50:44: Consumer storing message: 43 (size=2) 16:50:44: Consumer storing message: 37 (size=1) 16:50:44: Consumer storing message: 83 (size=0) 16:50:44: Consumer received event. Exiting

Thread object-lere(primitive-ler) esasen daxildir: Semaphore, Timer, Barrier. Semaphore (Atomic) - bir nov counter rolu oynayir. Acquire() time-da semaphore=0, release()-de =1. Timer - thread timer-de qoyuldugu andan sonra icra olunur. Bitirmek istesek, cancel() call olunur. User terefden gozlenilen activity olmasa, cancel() automatic call olunur. Barrier - mueyyen sayda thread-lerin start() olmasi uchun kemiyyet deyeri rolunu oynayir. Elbette ki, yene de OS scheduling-i unutmaq olmaz.

Sonda, GIL related ve thread-ler arasi emeliyyatlarin real-time tracing, logging, recording-i --> acquisitions, releases, conflicts, retries, etc:

Simple Trace: t2 100 5351 ENTRY t2 100 5351 ACQUIRE t2 100 5352 RELEASE t2 100 5352 ENTRY t2 100 5352 ACQUIRE t2 100 5353 RELEASE t1 100 5353 ACQUIRE t2 100 5353 ENTRY t2 38 5353 BUSY t1 100 5354 RELEASE t1 100 5354 ENTRY t1 100 5354 ACQUIRE t2 79 5354 RETRY t1 100 5355 RELEASE t1 100 5355 ENTRY t1 100 5355 ACQUIRE t2 73 5355 RETRY t1 100 5356 RELEASE t2 100 5356 ACQUIRE t1 100 5356 ENTRY t1 24 5356 BUSY t2 100 5357 RELEASE

Meqalede bir sira C/Python(Core) contributor-larin meqaleleri, forumlardan, github repolardan, seminarlardan, achiq istifadede olan saytlardan, eyni anda kitablardan ve demonstrasiya uchun kod numunelerinden(code examples), izahlardan faydalanilmish ve mueyyen hisselerde modifikasiya edilmishdir.

1 note · View note