ikxi

[ikxi]

...my terminal

[ikxi]

Pokkessembly

Just having the Intel Developer Manual for CPUs casually open on your computer is funny.

So many CPU instruction, so many pages

[ikxi]

Pokkepanic

WHY THE ***** WOULD IT DO THAT WHY WOULD A PRINT STATEMENT NOT BEING IN THE PROGRAM CAUSE IT TO CRASH????????????????

int get_token_amount(char *file_buffer, char *delimiters, int size) { char *internal_file_buffer; memcpy(internal_file_buffer, file_buffer, size + 1); char *save_prt; char *token = strtok_r(internal_file_buffer, delimiters, &save_prt); int token_count = 1; while (1) { printf("%s\n", token); token = strtok_r(NULL, delimiters, &save_prt); if (token == NULL || token == "\0") break; token_count++; } return token_count; }

THAT PRINTF NOT BEING THERE CAUSES A CRASH IN THE MEMCPY FUNCTION HOW IT COMES BEFORE THE PRINT STATEMENT HOW ARE THOSE TWO RELATED GCC WHAT DID YOU DO

[ikxi]

Pokkenizer

Finally finished my tokenizer in C and now I'm able to tokenize ANY text (as long as the delimiters are " \n\t\0") BUT BESIDES THAT, ALL

It's really funny, cause as a test file I use the source code, so the code is tokenizing itself kon so anytime I write something it just becomes more and more tokens

Next step is adding a way to split every more tokens like ()[]{}+-;."*/= but then have those token still be inside the file kon maybe I could do it by splitting each char, though then I'll have a problem with null-terminators, but ohh well, I will probably think of a solution

And it's all written *without* AI :3

#include <ctype.h> #include <stdio.h> #include <stdlib.h> #include <string.h> int get_token_amount(char *file_buffer, char *delimiters, int size) { char *internal_file_buffer; memcpy(internal_file_buffer, file_buffer, size + 1); char *save_prt; char *token = strtok_r(internal_file_buffer, delimiters, &save_prt); int token_count = 1; while (1) { printf(token); token = strtok_r(NULL, delimiters, &save_prt); if (token == NULL || token == "\0") break; token_count++; } return token_count; } void get_tokens(char *file_buffer, char *delimiters, char *token_buffer) { char *save_prt; char *token = strtok_r(file_buffer, delimiters, &save_prt); int string_position = (int)strlen(token) +1; for (int i = 0; i < string_position; i++) { token_buffer[i] = token[i]; } while (1) { printf(token); token = strtok_r(NULL, delimiters, &save_prt); if (token == NULL || token == "\0") break; int token_length = (int)strlen(token) + 1; for (int i = 0; i < token_length; i++) { token_buffer[i + string_position] = token[i]; } string_position += token_length; } } int main(int argc, const char * argv[]) { FILE *fptr = fopen(argv[1], "r"); if (fptr == NULL) return 1; fseek(fptr, 0, SEEK_END); long size = ftell(fptr); fseek(fptr, 0, SEEK_SET); char *file_buffer = malloc(size + 1); fread(file_buffer, size, 1, fptr); fclose(fptr); char *delimiters = " \n\t"; int token_count = get_token_amount(file_buffer, delimiters, size); char *token_buffer = malloc(size + 1 + token_count); get_tokens(file_buffer, delimiters, token_buffer); free(token_buffer); free(file_buffer); return 0; }

[ikxi]

POKKE

[ikxi]

Teeth brushing thought

judt before midnight, washing myself to get ready for bed, I think of a way to write a tokenizer in assembly. previously tried it and failed, but now i got a grasp on how I'd do it

WHY DO THESE THINGS ENTER MY BRAIN BEFORE I WANNA SLEEP

so like, i just read a string with the read syscall and then have the length in RAX, no need to make a convoluted get length function (though i could just make a for-loop that goes until an end-of-file signal, but i already have the length) and then i read and compare the ascii characters to stuff. if it's a number i continue reading and adding the numbers to a buffer until i encounter not a number and then it's probably an operator, so i save the number and then it's length in an array, i somehow gotta figure out how to make that array and then a way to signify its element, then i wouldn't have to save the size of the element, would be wasted memory. next i read the operator and store it also and then maybe there is a variable name which i then also store. basic tokenizer, and then i need to either build an AST or if it's just math, I can't do Shunting-Yard with reverse Polish notation

WYH AM I THINKING OF THIS JUST BEFORE SLEEP

[ikxi]

Pokke

I love Pokke

Pokke is great

[ikxi]

What?

How does this website work?