Commit Graph

158 Commits

Author SHA1 Message Date
shockrah
c850d42ce1 + Jwt tables - SEE NOTE
! wat - because have to do maintain permissions on a per request level we have
to do this check for permissions at what is basically every level, this does
mean we have to hit the database for a lot of routes however there is a check
that requests go through in order to avoid hitting the database whenever possible

+ rng field in claims now has real purpose
It's purpose is to act as a validator field in the jwt table. By verifying rng
fields we no  longer have to store whole jwt's
2021-05-08 01:29:44 -07:00
shockrah
9a22713080 + Testing /invite/join
+ Testing /meta
+ Testing /neighbors/list
2021-05-06 17:12:49 -07:00
shockrah
34115477ab + Mostly adding quality of life changes in this patch with the mock client
These changes will make it easier to read through verbose logs with some tactful
colors (yellow)
2021-05-06 17:03:24 -07:00
shockrah
181b1cadc4 + Exposing serde_json to db-lib lyaer
This dep was already there but just being used (as we're using serde). Now we can use it at the db layer which makes neighbor struct creation easier
* Cargo locks are being updated to reflect new dependancy changes
These changes actually reduce the dependancy count overall
+ Adding a very simple table to hold all the neighbors that an instance may want to reference
At some point we may want to support vanity join urls and this would be a good place to reference those
There would also need to be a way for admins to add/edit/remove vanity urls but that's for another patch
2021-05-06 13:29:32 -07:00
shockrah
a628d56e25 - Removing dotenv from required dependancies
+ Leveraging serde_json to use a config.json
This lets us use the same config for both the json-api and the rtc-server without adding dependancies
2021-05-05 15:50:39 -07:00
shockrah
f094ab49de * Adding some safey into the mock-client tester so that it doesn't just explode in pipelines 2021-05-03 13:24:24 -07:00
shockrah
7c8bc8b4fc - Removing warning
! Increasing wait time on client mock script to avoid firing too early
2021-05-01 19:16:18 -07:00
shockrah
34426038d9 * Fixing busted invite test in api client tester
+ Adding some more invite status checks
+ Adding a verbose invite usage check for sanity reasons

- Removing run-api-tests script as it now goes into the /scripts/ directory

+ Adding the wss-hmac setup from the command line arg
It was literally just 1 if statement that I forgot to write in
2021-04-27 13:27:39 -07:00
shockrah
61a3f1d4f5 + Adding first pass of a pair of invite endpoint tests 2021-04-26 02:01:04 -07:00
shockrah
c9658ad5b4 + Adding command line flag [-H/--hmac] to specify hmac file path
This is really just for testing purposes since the files tend to be in
awkward to reach path

- Removing if args.len == 0 check
Basically cannon-fodder tbh

* Auth now reads hmac path from environment var

! All of the above is added for the wss hmac as well

+ Adding command line flag [-W/--wss-hmac] to specify wss-hmac file path
2021-04-26 01:58:18 -07:00
shockrah
90b584eaa2 * Fixing /invites/join as it was returning the wrong struct data 🤦
Secret value given to user was literally the encrypted secret because I forgot
to rebind it

- removing dev key lmaoooo
These are generated super easily and this key doesn't actually go to anything
so its really not a problem, although it is annoying that its there
2021-04-25 12:40:08 -07:00
shockrah
e2e68c51ba - Removing old client test code in favor of new structure
+ Moving tests to the new client
- Removing web/ module
! Currently all tests are passing 17/17 but the real trickery comes with doing this
on CI which should will likely take some magic somewhere
Or we'll just extend the freechat docker image to finally have all the required
dependancies and just test on that with diesel and what not
2021-04-21 21:38:25 -07:00
shockrah
9ab9cdb176 + db-lib::Channel::get added and Returns a Result<Option<Channel>> for simplicity
This is really only for check if a particular channel exists for /messages/recent's backend

* Flattening db-lib::Message::last_n
SHould make code more readable as it now has a more linear flow

* api::Messages::recent_messages now 404's when a channel that doesn't exist is request
2021-04-21 21:33:30 -07:00
shockrah
432e94054f Fixing minor errors that were preventing tests fro mbeing ran oops :p 2021-04-21 17:23:00 -07:00
shockrah
38ff0edd39 ! Massive test suite overhaul see details below !
Problem: the old test suite was extremely inflexible
This meant that making new tests took way too much time.

+ This new rework makes the new client's backend much thinner and less "magical"
With less magic going on we can pass way more data more easily to the actual
http-request engine making the convenience wrapper over top it much more flexible

Translating old tests to the new engine might take a while but for now the old
client codebase is completely deprecated and will no longer be used+updated
2021-04-21 17:22:16 -07:00
shockrah
c4d7eb9111 * api::channels::list_channels has more relevant docs
Better matching against return types so we're not throwing away errors
	This just entails swapping the if let for a match statement

* db-lib::Message::send has more updated docs on what variants of Response it returns

* api::messages::send_message has better error logging now and more relevant comments
2021-04-21 17:14:33 -07:00
shockrah
ef73b39f4f + Adding db-lib::common::try_error_passthrough
This is more heavily used in Message::send as a way of discerning "real" server errors
from those that are caused  by user input. The name itself might not be super fitting either

* Moving code for inserting textual messages into its own function
This splits up the logic a bit for Message::send but this segment of logic is also
much simpler than that of file upload

* Flattening Message::send overall
Keeping this function as flat as can be is crucial as it is one of the heavier+most
important funcs in the whole JSON API codebase

Files are now also generated correctly and asynchronously
2021-04-21 16:08:23 -07:00
shockrah
52676cdd1f - Removing /messages/from_id as its no longer usable with the new id generation model
+ Adding new flags to sample .env file
+ Wrapper script run-api-tests.sh which does as its called
- Removing build.sh in favor of run-api-tests.sh
Makefile takes care of building, and apitests are ran with special script anyway
2021-04-20 12:09:05 -07:00
shockrah
bb2201c00e ! First pass base implementation for /messages/recent
This api method is made for clients to easily say "give me the n[1,100] latest
messages in some channel
This commit contains code that is largely untested but contains some base code
such that I can correct issues in the next _tested_ commit
2021-04-17 15:55:10 -07:00
shockrah
522890fa00 + rtc::new_message now takes a Pool ref to make the second call for the user name
! Start taking public user params like this in the JWT to reduce on network hits
It's cheap and reliable enough but the idea just now came to me so do that at some point
2021-04-14 22:46:31 -07:00
shockrah
89bd257213 * More explicit query string guard in /channels/list endpoint handler
* rtc::make_url now builds websocket urls more robustly incase we have to percent
encode them
2021-04-10 16:45:13 -07:00
shockrah
283c201e96 * More explicit query string guard in /channels/list endpoint handler
* rtc::make_url now builds websocket urls more robustly incase we have to percent
encode them
2021-04-10 16:44:24 -07:00
shockrah
368c9bc95f + Better logging in main
+
2021-04-07 21:06:08 -07:00
shockrah
d030accddc * Adding id to insertion call in db-lib::channel::add
This is done at the driver level because the member id's are randomly generated
u64's
2021-04-07 20:48:58 -07:00
shockrah
3f1dfbf824 Adding rtc notifications to relevant endpoints
! Still need a feature flag to compile this optionally instead of all the time
- Removed warning in main's auth::login_get_jwt call
2021-04-02 13:56:44 -07:00
shockrah
be54ce6a57 - Removing Claim::new since its used once
+ New notify driver function which does the network magic we need
+ Adding event emitters for the following events
"delete-channel"
"create-channel"
"update-nick"
2021-04-02 13:55:21 -07:00
shockrah
17f281c91c Removing websocket library for tokio-tungstenite (rtc) 2021-04-02 12:29:53 -07:00
shockrah
e496fbdd1a Message event now asynchonously emitted from /message/send to rtc server
+ Server claims are now also generated per request so this could get some benefit from caching the jwt
One idea is to cache a jwt for a temporary amount of time
+ event! macro also lets us generate json payloads more easily
! Still requiring this module to built under an optional feature however that should be done easily
2021-04-02 12:11:59 -07:00
shockrah
c6a49a8437 User JWT's now have nbf field in claims set in seconds 2021-04-02 12:06:17 -07:00
shockrah
afdeef0a49 More warning removals and more db-lib result wrapping
This should be everything now
2021-03-30 23:00:50 -07:00
shockrah
4615374357 Comfy dev scripts to make life a little bit easier 2021-03-30 22:49:46 -07:00
shockrah
c45f8d0482 More db-lib ok/err wrapping
Some misc changes that don't really belong with anything else are bundled here
Those changes aren't actually very important however
2021-03-30 22:46:59 -07:00
shockrah
75dcb7b73e No more jwt db checks
Also wrapping relevant db code with ok/err checks
2021-03-30 22:45:39 -07:00
shockrah
41202507c1 removing warning 2021-03-30 22:44:09 -07:00
shockrah
4cbbfdd7a2 mysql_async updates
* More shorthand usage for clearer code in invite create
2021-03-30 22:43:07 -07:00
shockrah
54f4dd53be - Removing non-result returns
+ Updating further for newest version of mysql_async
2021-03-30 22:42:34 -07:00
shockrah
fea94bffa0 Removing features flag from db-lib as its not needed 2021-03-30 21:41:57 -07:00
shockrah
15b56353b8 Removing passthrough for message send handler
This means we get more opaque errors for the clients but it also results in way less code to maintain
2021-03-30 21:39:44 -07:00
shockrah
d88948385d Upgrading mysql_async to latest version
This version comes with the benefit of having much more concise exec_map functionality among other things

This commit and the following ones also move db-lib to using a more 'try' based approach
2021-03-30 21:38:16 -07:00
shockrah
f7d90e4a09 - Removal of auth code in db-lib 2021-03-30 21:36:21 -07:00
shockrah
5bbc57313f + Json-API now pushes proper new-message event to rtc server correctly
+ Rtc server takes in new-message correctly but with basically no ux/auth for now
This should be fine as we can layer on security afterwards with no issue
2021-03-25 20:46:10 -07:00
shockrah
52fab01119 + Clearer logging from the json api
* Renamed db helper function to be (debatably) more clear in its intentions
	That function and everything in that module is quickly become awful however

+ Adding some notes for later db-lib development
2021-03-24 01:43:52 -07:00
shockrah
a1f86fdf6e * Moving away from auto_increment
! Initial id size will stay as u64 but u128 are easily the next major change
This change shouldn't break clients written in high level languages since most already use 128's under the hood anyway
- This commit also removes the auto_increment flag from basically everything that uses RNG id's
2021-03-24 01:14:37 -07:00
shockrah
d02084a22c + Adding proper file upload support
! Requirements for variable message fetching not yet met
! /message/get?id=<id> needs a way of passing back file contents
! The file upload is sketchy at best and not necessarily guaranteed to sync the
database with the file system
2021-03-23 17:16:05 -07:00
shockrah
975acfd606 Backend REST API changes:
* Fixed weird match with get_online_members
+ Now using special(smoll) Public Member struct for public member data fetches as.
They're size on networks should be pretty small so we can package a ton of them in a single request

DOCS changes:
- Removing references to unix timestamps in seconds
- Removing references to joindate
* Exact content-type values now specified
2021-03-20 22:44:22 -07:00
shockrah
03c41b6833 - Removed joindate from members schema
+ Auth module now uses std::time for time based calculations
! All time notations are i64 and rounded to the Millisecond

* Moving db pool to a lazy static to avoid constructing a whole pool on every request

+ Adding more logging per request, even if its lazy logging
* Content-Types are now correctly written per type
2021-03-20 19:47:59 -07:00
shockrah
988aa9f155 Removal of chrono and async_trait as dependancy in db-lib
Removig chrono from api code as well

Removing chrono as dep in api code
+ Using Content-Type for /message/send content type

Updating cargo lists for removal of fluff deps

Removal of more fluff

Addking makefile to avoid compiling debug builds by accident while developing
2021-03-20 14:59:06 -07:00
shockrah
9ce04e96a7 Query string parameter 'type' is now enforced by the api
+ Flag is literally called 'type'
2021-03-12 02:17:06 -08:00
shockrah
ddb08818b1 Updating schemas to support content_type 2021-03-09 00:03:12 -08:00
shockrah
ae675d000b content_type now a supported column in db-lib
 API layer now behaves as expected, returning 'content_type' flag with each message
2021-03-09 00:02:20 -08:00
shockrah
7125fc954d SERVER_PROTOCOL is now an official env variable for the json-api to read 2021-03-07 23:55:38 -08:00
shockrah
5fc3e2a553 /channels/create refactored to use the new db-lib api
 Reduced branching a shit ton

Testing was ass but it should pass client tests now, back to normal
2021-03-07 17:47:24 -08:00
shockrah
7c3537e4f6 Updated target for debugger
 Tests now use the new flags required for /channels/create
 Doubled size for channel descriptions
2021-03-07 17:46:17 -08:00
shockrah
fadc7d6dc1 Removing more unused trait methods in db-lib's channel module
 /channels/list now takes a "type" parameter which defaults to TEXT_CHANNEL(1)
 Refactoring db::Channel::filter to use a more latency friendly approach
 db::Channel::Filter now returns Result<Response<Self>, SqlError>
2021-03-07 14:26:50 -08:00
shockrah
12936d5b1a Removing double log issue with client-tester
 Moving Messages struct in db-lib to root for less namespacing uglyness
 Removing an whole unused trait
 Message::get_time_range now returns a special 'UserMessage' struct that includes the name of the author
 More code must be removed from db-lib since a ton of the trait methods aren't used anymore
2021-03-07 13:29:10 -08:00
shockrah
a6e3f22eae Reverting package upgrades 2021-03-07 12:43:25 -08:00
shockrah
2fa77fecb1 Updating hyper and tokio to newest versions for better http2 support 2021-03-05 22:59:27 -08:00
shockrah
898a7a8ca2 New /members/single route
 Simple test for /members/single handler
 Should probably add some more tests to verify failure cases
2021-03-05 19:23:50 -08:00
shockrah
b999907540 * channels are now nested in "channels" key 2021-03-03 16:30:39 -08:00
shockrah
05acbfa94e Passing tests for invite routes 2021-02-25 16:05:46 -08:00
shockrah
212173f543 Fixing /join route, time check was backwards oops and it was set to GET not post 2021-02-25 15:31:45 -08:00
shockrah
ab1c9e40f7 Ower creation now gives back prettier output + server meta vars as well 2021-02-25 14:03:34 -08:00
shockrah
64125a7874 dumy test stats for no reason 2021-02-21 20:10:14 -08:00
shockrah
93fd8c2cd6 +Adding new tests for /members/me/nickname
* Fixed incorrect param key in endpoint handler
+Adding a ycm conf for easier writing
2021-02-21 18:52:35 -08:00
shockrah
5db976b9de + New route for /members/me/nickname
! No testing yet that comes in next patch, with any required fixes
+ db now has specific method for Member::update_nickk
2021-02-21 17:21:17 -08:00
shockrah
7263df8928 Adding new routes for /members/get_online & /members/me
Passing the previous tests as well
2021-02-20 20:15:43 -08:00
shockrah
34736bef3f updating tests for members routes 2021-02-20 20:14:10 -08:00
shockrah
c896870397 Reformatting of wiki:
* Adding endpoint docs and correcting some typesetting mistakes
* Change qs params to match documentation
* Splitting up articles to geneate the sidebar links properly'
* Bad baseurl issue fixed in previous patch
2021-02-18 16:29:29 -08:00
shockrah
71ebefce51 Dummy commit 2021-02-13 19:08:21 -08:00
shockrah
da1b4c3253 Trivial change to test out new filtered pipeline 2021-02-13 19:00:22 -08:00
shockrah
4323156242 Building and testing against proper nginx setup now
! No longer pipeline ready because the docker image is out of date with this change
2021-02-11 22:28:20 -08:00
shockrah
9e70e7272b removing busted conf 2021-02-11 20:21:30 -08:00
shockrah
6bc8dd0d7d tests are now ran with https as we now use https with nginx as a frontend however:
The client tester is super scuffed and no longer in a pipeline ready state as a new docker image has to be created(again)
2021-02-11 20:06:56 -08:00
shockrah
fb36f70d95 Response logs now optionally show the query string when logging, defaulting to always show the logs for now however 2021-02-11 15:04:41 -08:00
shockrah
ab6b8e460b VSCodium python interpreter directory change 2021-02-10 23:21:11 -08:00
shockrah
25755bf394 Percent decoder func found on crates.io
Link: https://docs.rs/urldecode/0.1.1/src/urldecode/lib.rs.html#1-21

Why rob? Well its slightly modified(very smol change) in that it now takes a &str and allocates a string from it
Later patches will change this so that we don't _always_ allocate frivolously
2021-02-10 23:20:51 -08:00
shockrah
16276c97e0 Moving option parameters to the query string
Large payloads will remain the body as those are typically required for post endpoints such as /message/send
2021-02-10 23:18:24 -08:00
shockrah
286a77d9ac Using virtualenv python interpreter from now on 2021-02-10 23:16:41 -08:00
shockrah
f585cf122c * switching to building options in query string
* payloads for things like /message/send are now send through the request body
* no longer sending body data in gets

This whole patch is basically so we can allow js clients to exist btw -_-
2021-02-10 23:16:19 -08:00
shockrah
21b184b324 Client tester now uses channel_id where applicable for api parameters
as well as being less explicit about what it passes to the query string
2021-02-10 23:15:05 -08:00
shockrah
2d6739438a removing unused redis dependancy 2021-02-10 18:20:28 -08:00
shockrah
eb338e03a0 + Test for qs_param behavior
Pretty simple and contains a special note about qs_param usage
2021-02-05 16:57:31 -08:00
shockrah
769aa72cdf Opting for qs_param to reduce on visual cancer 2021-02-05 16:39:24 -08:00
shockrah
ab9fef2ccc route_dispatcher now reflects new hashmap usage over serde_json::Value's
HashMap of query string parameters is generated in main_responder before passed on
2021-02-05 16:32:03 -08:00
shockrah
33ae768ae4 *+ More qs_param! usage 2021-02-05 16:29:32 -08:00
shockrah
3369b4ec48 - Removing unused code 'process_expires_parameter' function
* Switching to qs_param in only a few cases for now
2021-02-05 16:28:47 -08:00
shockrah
2cb2d2b29e + Adding serialize to db::Invite type
* Invite initialization using shorthand notation now
2021-02-05 16:26:43 -08:00
shockrah
ba5db17706 * Replacing extract_uid with qs_param!(...) usage 2021-02-05 16:25:35 -08:00
shockrah
4c92b20e74 * replacing extract_uid usage with qs_param!(....).unwrap() usage 2021-02-05 16:25:12 -08:00
shockrah
8e98df1d37 - Removed parse_json_params as its no longer used
+ Adding qs_param macro to remove boilerplate in pulling out Option<T> data from hashmaps
2021-02-05 16:24:39 -08:00
shockrah
c2e384a13a */message/send
Now polls content from the request body
* flattened a bunch of checks with data fetches to the db
Outright reduction of branches yey
2021-02-04 01:58:57 -08:00
shockrah
20aca8a069 +Parsing channel_id param correctly
+ Checking for proper permissions in user

moar if let bindings!!!1
2021-02-03 23:29:53 -08:00
shockrah
42e783ccf1 Clarification on failure return value in delete trait method 2021-02-03 23:28:14 -08:00
shockrah
7334bb287d renamed add to 'new' to show its _real_ intentions 2021-02-03 21:56:34 -08:00
shockrah
44b51133d2 * /channels/create handler uses hashmaps
* cleaned up wall of parameter parsing to be less cancerous(read diff to die instantly)
- Removed repeated db::Response in response mutator match

 Simplified how parameters are parsed by making them actually readable
2021-02-03 21:56:09 -08:00
shockrah
c1ef4c6336 -Removed _most_ direct sql manipulation for invite creation
- Removed unused/irrelevant unit tests, which now have to be rebuilt for the new codebase
2021-02-03 21:26:26 -08:00
shockrah
39a4d2a247 +New .add method for db::Invites
+New new() method to reduce on eye clutter
2021-02-03 21:25:27 -08:00
shockrah
9fac3aa117 * /join route handler now uses hashmaps among and returns more sensible http codes 2021-02-03 21:09:12 -08:00
shockrah
98803eec26 First pass of using hashmaps in /invite/create handler
+ Adding allowed_perm_invite, a helper function pulling permissions form the database to check if the user can make invites
2021-02-03 20:50:06 -08:00
shockrah
45af62ceb3 Inlining helper func for permission checking 2021-02-03 19:31:38 -08:00