chrizandr-blog

[chrizandr-blog]

Week 8 – GSoC

Another revamp for Hydrus!

So another major part of Hydrus got scrapped and reconstructed. The DB connectors we were using(SQLAlchemy) were defined in the application and when the app runs, the connection is established through the connector defined in Hydrus. There wasn’t much we could do about that and no actual way to change it during runtime, this was a problem mainly for writing tests. When testing the app, any data that was added was pushed directly to the main database, because of the connector.

Earlier,  we had isolated the data operations from the actual server by creating abstract methods in crud.py. The application would just plug in the connector to these operations and do the work. We could easily test the app(without doing any operations) and the crud.py operations using another connector.

Problem? The improved design made the operations and the API Doc largely dependant on each other. We had to check if the documentation allowed endpoints/operation and also the type of data that was being sent in the request. There was no way of checking how these things worked together.

Solution? App contexts I had to look for solutions for this problem and came across something known as an app context in Flask. This was basically an object called flask.g that allowed you to store stuff in the app objects itself and use them based on the context in which the app was used.

Simple example: You use one connector when the actual app is running(main database), and you can swap out the connector when testing for another one(test database). I know flask has built in support for most of this, but since we never actually used the ORM from Flask, it was pretty cool to implement this functionality on my own.

So, new commits and 10 new tests added :). [commit]

The good news is that all of them passed without any problems.

Another cool thing: Hydrus now actually reads the API Documentation and creates objects based on the specification mentioned in the doc and then runs tests using those objects. So no matter what API Doc is there on the app, the tests will work for them all

[chrizandr-blog]

Week 7 – GSoC(pt.2)

And Hydrus just had a total makeover

Here’s the commit.

Everything uses the API Doc writer. Contexts are generated when classes are added in the beginning. EntryPoint is created and stored in the HydraDoc object as an EntryPoint object. All classes are stored as HydraClass objects, if specified a collection for each class is created as HydraCollection. All operations are stored as HydraClassOp in the classes and EntrypointOp in the entrypoint. Supported properties for classes are stored as HydraClassProp objects. All in all, I feel good adding this to Hydrus, way more efficient than what we had before.

Also made a few changes to Hydrus design, will improve things and add tests soon for Hydrus.

Meanwhile, Akshay is working on the simulation part, when Hydrus is ready, it should only be a matter of plug and play

[chrizandr-blog]

Week 7 – GSoC

Finished Doc writer

Here’s the Commit.

So now, we have this awesome abstraction that can create Hydra API Docs, generate EntryPoints, Contexts and a bunch of other stuff.

Now that this is done, I want to add this to Hydrus. Right now, Hydrus handles the docs part in a very messy way, most of the stuff is generated for every request from something called parsed_classes. Akshay had added this during the first stage to get things working, but it’s high time we have an abstraction for it.

Also changed the API Doc for the simulation some more. All of it is on the writer branch in https://github.com/HTTP-APIs/hydra-flock-demo.

More updates when I add this to Hydrus.

[chrizandr-blog]

Week 6 – GSoC

So we have distributed our work for now. I’m writing the API Doc for the simulation while Akshay works on the simulation implementation. We have a basic design in mind, this is subject to change in the coming days, but will be mostly followed:

Basically the endpoints can be independently setup using just an API Documentation and Hydrus. Once that is done, the controllers will use the endpoints to do the simulation. This will keep the REST API separate for the simulation and will also showcase how we can create an API just by using the Hydra API Doc. Hydrus needs to be worked upon a lot. I also wrote a couple of gists for the functions that may be needed on the drones for simulation. Here’s one for the server: .codebox { /* Below are styles for the codebox (not the code itself) */ border:1px solid black; background-color:#E8E8E8; width:100%; overflow:auto; padding:10px; } .codebox code { /* Styles in here affect the text of the codebox */ font-size:0.7em; /* You could also put all sorts of styling here, like different font, color, underline, etc. for the code. */ font-family:Consolas,Monaco,Lucida Console,Liberation Mono,DejaVu Sans Mono,Bitstream Vera Sans Mono,Courier New; }

"""Pseudo code for drone control server.""" class Server(): def __init__(self, drones): now = time() self.drones = dict() for drone in drones: self.drones[drone.id] = drone self.log = [LogEntry("Created server at %s" % (now), now)] self.data = list() def retrieve_drone_list(self): now = time() event = LogEntry("All Drones requested at %s" % (now), now) self.log.append(event) return self.drones def retrieve_position(self, id_): now = time() event = LogEntry("Drone %s position requested at %s" % (id_,now), now) self.log.append(event) return self.drones[id_].position def retrieve_speed(self, id_): now = time() event = LogEntry("Drone %s speed requested at %s" % (id_,now), now) self.log.append(event) return self.drones[id_].speed def retrieve_battery(self, id_): now = time() event = LogEntry("Drone %s battery status requested at %s" % (id_,now), now) self.log.append(event) return self.drones[id_].battery def retrieve_drone_current_action(self, id_): now = time() event = LogEntry("Drone %s status requested at %s" % (id_,now), now) self.log.append(event) return self.drones[id_].status def change_position(self, id_, position): now = time() event = LogEntry("Drone %s position changed at %s" % (id_,now), now) self.log.append(event) return self.drones[id_].update_drone_position(position) def change_speed(self, id_, speed): now = time() event = LogEntry("Drone %s speed changed at %s" % (id_,now), now) self.log.append(event) return self.drones[id_].update_drone_speed(speed) def change_status(self, id_, status): now = time() event = LogEntry("Drone %s status changed at %s" % (id_,now), now) self.log.append(event) return self.drones[id_].update_drone_status(status) def log_data(self, id_, data): now = time() event = LogEntry("Drone %s status changed at %s" % (id_,now), now) self.data.append(data) def coordinate(self, drones): now = time() event = LogEntry("Drone coordinated at %s" % (now), now) for drone in drones: # Run some algorithm drone.position = new_position def recharge_drone(self, id_, charge_time, rate, power_source): elapsed = time() now = time() self.drones[id_].recharge(power_source, rate) while now - elapsed < charge_time: now = time() self.drones[id_].discharge()

And one for the drone:

"""Pseudo code for the drones.""" class Drone(): def __init__(self, position, speed, model, status, id_, server): self.position = { "latitude": position["latitude"], "longitude": position["longitude"] } self.speed = speed self.battery = 100 self.status = status self.id = id_ self.model = model self.control = server self.data = list() def update_drone_status(self, status): self.status = status return "okay" def update_drone_position(self, position): self.position = position return "okay" def update_drone_speed(self, speed): self.speed = speed # Needs to be started as a process, raise some error/exception def discharge(self, rate): current = time() while drone.status == "active": now = time() if now - current > rate: self.battery = self.battery - 1 current = now if self.battery == 0: raise BatteryEmpty() def update_drone_status(self): status = self.control.send(self.id, self.status) return status # Needs to be started as a process, raise some error/exception def recharge(self, power_source, rate): # Need to stop the discharge process first # .... # .... # Start recharge process current = time() while power_source.connected(): now = time() if now - current > rate: self.battery = self.battery + 1 current = now if self.battery == 100: raise BatteryFull() raise BatteryCharged() def get_nearby_drones(self, radius): drones = self.control.get_nearby_drone(self.position, radius) return drones # Needs to be started as a process too def coordinate(self, radius) drones = self.get_nearby_drones(radius) self.control.coordinate(drones) def get_drone_position(self): return self.position def get_drone_status(self): return self.status def get_drone_speed(self): return self.speed def get_drone_battery(self): return self.battery # Needs to be started as a process def collect_data(self, threshold): current = time() while self.status == "active" and self.battery > 0: now = time() if now - current > threshold: data = Data(self.sensor.read(), now) self.control.log_data(data) current = now()

I’ve also started writing a doc_writer, for easy creation of a Hydra API Doc, right now it’s just some functions that arrange things into Python dicts, but I’m planning to create classes for the different parts of the documentation so that we can create the Documentation, EntryPoint, Context and other docs on the fly. Here is the initial version, let’s see where this will go. This commit also contains the API Docs I designed for the central-server and drone. That’s it for now 🙂

[chrizandr-blog]

Week 5 – GSoC

So first evaluation is done. Thankfully I’ve passed

Now to the next step of the project. We had a discussion with our mentors and have decided to use Hydrus and create a simulation of drones to showcase how Hydra can be used in a real world scenario. We decided that Space and Subsystem demo would be too complex and it’s better to stick to something easier.

For now, we’re just working out the complexities of the problem. Making endpoints that will be needed for the simulation and other things.

Our idea is mainly discussed in this Google Doc.

I’m gonna start writing an API documentation for this design and see if we can speed up the process. If we have a good enough API doc, Hydrus should be able to use it to have a server with endpoints ready.

After that, we can do the simulation part later.

let's hope for the best

[chrizandr-blog]

Week 3 GSoC

Just a few days left before Phase-1. Nervous as hell :P

This week was mostly testing and documentation.

Akshay got a server up and running for serving data, he also linked them to the CRUD operations. He also created a docker container for easier deployment. That guy is pretty good. You should check out his blog too: https://xadahiya.github.io/

Akshay’s server wasn’t dynamically serving data at the class URL /api/<ClassName>. I implemented dynamic views for each class. Made changes to CRUD to return proper HTTP Response codes. Had to do a major makeover of CRUD to check so many different cases. Damn hectic

I wrote tests this week for the CRUD Operations.

Also wrote tests for the server.

Modified the contexts for the data that the API was serving and created dynamic contexts to be generated on a separate URL.

Everything seems ready for the evaluation. The server is up and running, all tests are passing and we have an automatic docker that sets everything up.

Just need to update the wiki and deploy the server online.

That’s it for now. Cheers :)

[chrizandr-blog]

Week 2 – GSoC

So this week, we had a couple of things up and running. My partner Akshay(he’s also working with me on this project) has been able to create a parser for OWL Classes and properties and inserted them into the database.

We had a random data generator ready and we started entering data into our Graph. I implemented CRUD operations for the data to create/update/retreive/delete the data.

We decided on a data format and got things running. CRUD operations are working initially. Alshay will set up a server for serving the data we entered. Let’s see where this takes us.

I’ve also started working on a APIDoc parser for the server. Basically, a user can give two things to setup a server:

He can use an APIDocumentation to create a server.

Or if he doesn’t have the APIDocumentation, he can use an OWL/RDF vocabulary to generate one:

Also, I got a wiki up and running on https://github.com/HTTP-APIs/hydrus/wiki

Documentation is hard :P

[chrizandr-blog]

Week 1 – GSoC

This week coding has officially started. We discussed on the database models as to how data must be stored in PSQL.

Since we have to store a graph database, it’s quite a tricky problem.

We discussed and decided on the following DB Structure:

It might be a little difficult to understand, but I implemented it like this:

Which may be even more difficult to understand :P

But yeah, apart from this, I also implemented a parser that will take any OWL/RDF Json-LD file and generate a Hydra APIDocumentation for it. It was a bit trcky generating supportedProperty for each Hydra class, since OWL doesn’t explicitly mention how the property needs to be used, it only defines them. But with a little thinking, I figured how to check the domain and ranges for OWL Properties and Also restrictions for every OWL class to generate mappings between a Class and a Property.

Everything working for now. Let’s hope I can get some tests going for the same.

Cheers for now.

[chrizandr-blog]

Well, looks like I made it.

The road to GSOC, sounded like a nice title, but I settled for this instead. Hi readers, let me start by introducing myself. I'm Chris Andrew, you're average, everyday, 21 year old, computer science undergrad. I'm from India, the so called land of diversity, festivals and culture

I'm starting my GSOC journey with the Python Software Foundation, under the sub-org of the Hydra W3C Group. We at Hydra are an elite group of beings that aim to create a better internet for the world :P . We aim to create 'smart APIs' that can communicate with 'smart clients' to help automate most of the things that people have to code manually to work with APIs. For our noble cause, we have decided to use Hydra. It is a specific type of JSON Linked Data representation that was proposed by the W3C(They are also backing us in this project).  Heres a little more formal definition for those of you who are interested.

Introduction The Hydra draft as proposed here, describes a vocabulary to help servers and clients interact with each other and enable machines to use linked data for understanding semantics of the content they serve. The proposed server has an HTTP based endpoint for exchanging information with the client. The client uses this end point to understand the state of the server and also to modify this state. The methods classes, properties and operation that the server supports and that is authorized for the client to do is exposed by Hydra’s API documentation. This documentation is exposed at a standard URL, allowing clients to detect Hydra enabled servers.

Proposal For demonstration purposes, as proposed, I support the use of the Spacecraft and SubSystems vocabulary as mentioned in here. The implementation required for the demo would mainly consists of three parts:

A client that can understand Hydra vocabulary and interacts with a Hydra supporting server to retrieve needed information and modify existing information.

A server that can serve data and metadata(in the for API documentation) to the client over HTTP.

A middleware that allows users to use the client to interact with the server using Natural Language which is processed into Hydra based vocabulary to make it machine understandable.  

A simple example explaining the use of the above architecture would be:

User types in the query “What is the distance between Mars and Earth?”.

Middleware uses NLP to extract keywords ‘Mars’, ‘Earth’ and ‘distance’ and maps it to the Hydra instances and properties present at the server.(Maybe use word2vec with LSTMs for keyword extraction - tentative)

Middleware passes these instances and the underlying query to the client.

Client uses the API endpoints to extract the given information from the server. If possible client stores the queries for further reference(although ideally a client should only store state information and metadata).

Server replies with the required value.

I'd rather not overcomplicate things for now. But that is the gist of the things I aim to accomplish in this year's GSOC.