Commit Graph

88 Commits

Author SHA1 Message Date
Kegsay
1ad7219e4b
Implement /sync limited and read timeline limit from stored filters (#1168)
* Move filter table to syncapi where it is used

* Implement /sync `limited` and read timeline limit from stored filters

We now fully handle `room.timeline.limit` filters (in-line + stored) and
return the right value for `limited` syncs.

* Update whitelist

* Default to the default timeline limit if it's unset, also strip the extra event correctly

* Update whitelist
2020-06-26 15:34:41 +01:00
Kegsay
e560619f76
Refactor SendMembership - make ban test pass (#1160)
* Refactor SendMembership - make ban test pass

* Only check invite auth events for local invites
2020-06-24 18:19:54 +01:00
Kegsay
0577bfca55
Pass join errors through internal API boundaries (#1157)
* Pass join errors through internal API boundaries

Required for certain invite sytests. We will need to think of a
better way of handling this going forwards.

* Include m.room.avatar in stripped state; handle trailing slashes when GETing state events

* Update whitelist

* Update whitelist
2020-06-24 09:59:59 +01:00
Neil Alexander
dc0bac85d5
Refactor account data (#1150)
* Refactor account data

* Tweak database fetching

* Tweaks

* Restore syncProducer notification

* Various tweaks, update tag behaviour

* Fix initial sync
2020-06-18 18:36:03 +01:00
Kegsay
8efeb8eb3b
Return the correct /joined_members response and allow ?format=event (#1146) 2020-06-17 16:21:42 +01:00
Kegsay
e09d24e732
Move account/device DBs to userapi (#1141) 2020-06-17 12:05:56 +01:00
Kegsay
a66a3b830c
Make userapi control account creation entirely (#1139)
This makes a chokepoint with which we can finally fix
'database is locked' errors on sqlite during account creation
2020-06-17 11:22:26 +01:00
Kegsay
9c77022513
Make userapi responsible for checking access tokens (#1133)
* Make userapi responsible for checking access tokens

There's still plenty of dependencies on account/device DBs, but this
is a start. This is a breaking change as it adds a required config
value `listen.user_api`.

* Cleanup

* Review comments and test fix
2020-06-16 14:10:55 +01:00
Kegsay
0ba1245a46
Current wiring (#1125)
* Current wiring

* Add ServerKeyAPI lines
2020-06-15 10:13:57 +01:00
Kegsay
ecd7accbad
Rehuffle where things are in the internal package (#1122)
renamed:    internal/eventcontent.go -> internal/eventutil/eventcontent.go
	renamed:    internal/events.go -> internal/eventutil/events.go
	renamed:    internal/types.go -> internal/eventutil/types.go
	renamed:    internal/http/http.go -> internal/httputil/http.go
	renamed:    internal/httpapi.go -> internal/httputil/httpapi.go
	renamed:    internal/httpapi_test.go -> internal/httputil/httpapi_test.go
	renamed:    internal/httpapis/paths.go -> internal/httputil/paths.go
	renamed:    internal/routing.go -> internal/httputil/routing.go
	renamed:    internal/basecomponent/base.go -> internal/setup/base.go
	renamed:    internal/basecomponent/flags.go -> internal/setup/flags.go
	renamed:    internal/partition_offset_table.go -> internal/sqlutil/partition_offset_table.go
	renamed:    internal/postgres.go -> internal/sqlutil/postgres.go
	renamed:    internal/postgres_wasm.go -> internal/sqlutil/postgres_wasm.go
	renamed:    internal/sql.go -> internal/sqlutil/sql.go
2020-06-12 14:55:57 +01:00
Kegsay
b7187a9a35
Remove clientapi producers which aren't actually producers (#1111)
* Remove clientapi producers which aren't actually producers

They are actually just convenience wrappers around the internal APIs
for roomserver/eduserver. Move their logic to their respective `api`
packages and call them directly.

* Remove TODO

* unbreak ygg
2020-06-10 12:17:54 +01:00
Kegsay
98cb0705ea
Remove unused UserUpdates producer (#1109) 2020-06-10 10:34:22 +01:00
Neil Alexander
a5d822004d
Send-to-device support (#1072)
* Groundwork for send-to-device messaging

* Update sample config

* Add unstable routing for now

* Send to device consumer in sync API

* Start the send-to-device consumer

* fix indentation in dendrite-config.yaml

* Create send-to-device database tables, other tweaks

* Add some logic for send-to-device messages, add them into sync stream

* Handle incoming send-to-device messages, count them with EDU stream pos

* Undo changes to test

* pq.Array

* Fix sync

* Logging

* Fix a couple of transaction things, fix client API

* Add send-to-device test, hopefully fix bugs

* Comments

* Refactor a bit

* Fix schema

* Fix queries

* Debug logging

* Fix storing and retrieving of send-to-device messages

* Try to avoid database locks

* Update sync position

* Use latest sync position

* Jiggle about sync a bit

* Fix tests

* Break out the retrieval from the update/delete behaviour

* Comments

* nolint on getResponseWithPDUsForCompleteSync

* Try to line up sync tokens again

* Implement wildcard

* Add all send-to-device tests to whitelist, what could possibly go wrong?

* Only care about wildcard when targeted locally

* Deduplicate transactions

* Handle tokens properly, return immediately if waiting send-to-device messages

* Fix sync

* Update sytest-whitelist

* Fix copyright notice (need to do more of this)

* Comments, copyrights

* Return errors from Do, fix dendritejs

* Review comments

* Comments

* Constructor for TransactionWriter

* defletions

* Update gomatrixserverlib, sytest-blacklist
2020-06-01 17:50:19 +01:00
Neil Alexander
fe82e1f725
Separate muxes for public and internal APIs (#1056)
* Separate muxes for public and internal APIs

* Update client-api-proxy and federation-api-proxy so they don't add /api to the path

* Tidy up

* Consistent HTTP setup

* Set up prefixes properly
2020-05-22 11:43:17 +01:00
Kegsay
24d8df664c
Fix #897 and shuffle directory around (#1054)
* Fix #897 and shuffle directory around

* Update find-lint

* goimports

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-05-21 14:40:13 +01:00
Neil Alexander
334680a216
Get profile content for join request again (#1027) 2020-05-13 14:53:25 +01:00
Neil Alexander
9b1b095b49
Roomserver perform leave (#1004)
* First pass at PerformLeave

* Fix SQLite bulkSelectEventStateKey

* Update gomatrixserverlib

* Fix bugs

* Tidy a bit

* Satisfy King Linter

* Review comments

* Review comments

* Fix constants in SQLite event state keys table
2020-05-04 18:34:09 +01:00
Neil Alexander
5c894efd0e
Roomserver perform join (#1001)
* Add PerformJoin template

* Try roomserver perform join

* Send correct server name to FS API

* Pass through content, try to handle multiple server names

* Fix local server checks

* Don't refer to non-existent error

* Add directory lookups of aliases

* Remove unneeded parameters

* Don't repeat join events into the roomserver

* Unmarshal the content, that would help

* Check if the user is already in the room in the fedeationapi too

* Return incompatible room version error

* Use Membership, don't try more servers than needed

* Review comments, make FS API take list of servernames, dedupe them, break out of loop properly on success

* Tweaks
2020-05-04 13:53:47 +01:00
Neil Alexander
e15f6676ac
Consolidation of roomserver APIs (#994)
* Consolidation of roomserver APIs

* Comment out alias tests for now, they are broken

* Wire AS API into roomserver again

* Roomserver didn't take asAPI param before so return to that

* Prevent roomserver asking AS API for alias info

* Rename some files

* Remove alias_test, incoherent tests and unwanted appservice integration

* Remove FS API inject on syncapi component
2020-05-01 10:48:17 +01:00
Neil Alexander
64e94e9a6f
Join room support in federation sender (#989)
* Implement PerformJoinRequest

* Rename perform functions

* Check send join response

* Temporary wiring to test federation sender room joins

* Actually pass through the config

* Make sure membership content shows join
2020-04-29 15:29:39 +01:00
Neil Alexander
a308e61331
Federation sender API remodel (#988)
* Define an input API for the federationsender

* Wiring for rooomserver input API and federation sender input API

* Whoops, commit common too

* Merge input API into query API

* Rename FederationSenderQueryAPI to FederationSenderInternalAPI

* Fix dendritejs

* Rename Input to Perform

* Fix a couple of inputs -> performs

* Remove needless storage interface, add comments
2020-04-29 11:34:31 +01:00
Neil Alexander
895a72b6ee
Move /room/{roomID}/state endpoints into client API (#606) (#962)
* Move /room/{roomID}/state endpoints into client API (#606)

* Update sytest-whitelist

* Blacklist tests which rely on endpoints we don't implement
2020-04-14 18:36:08 +01:00
Kegsay
11a8059bba
Rename the typing server to EDU server (#948)
* Blunt move and sed rename

* Rename common/ refs to typing

* Rename internal stuff in eduserver

* Rename docs and scripts

* Rename constants/filenames, goimports everything to re-order imports
2020-03-30 15:02:20 +01:00
Kegsay
bfbf96eec9
p2p: Implement published rooms (#923)
* Create and glue ExternalPublicRoomsProvider into the public rooms component

This is how we will link p2p stuff to dendrite proper.

* Use gmsl structs rather than our own

* Implement federated public rooms

- Make thirdparty endpoint r0 so riot-web loads the public room list

* Typo

* Missing callsites
2020-03-19 11:04:08 +00:00
Prateek Sachan
dc06c69887
Implement /joined_rooms (#911)
* Implemented /joined_rooms

* Removed account endpoint added by mistake

* trigger ci
2020-03-19 10:25:36 +00:00
Kegsay
b6ea1bc67a
Support sqlite in addition to postgres (#869)
* Move current work into single branch

* Initial massaging of clientapi etc (not working yet)

* Interfaces for accounts/devices databases

* Duplicate postgres package for sqlite3 (no changes made to it yet)

* Some keydb, accountdb, devicedb, common partition fixes, some more syncapi tweaking

* Fix accounts DB, device DB

* Update naffka dependency for SQLite

* Naffka SQLite

* Update naffka to latest master

* SQLite support for federationsender

* Mostly not-bad support for SQLite in syncapi (although there are problems where lots of events get classed incorrectly as backward extremities, probably because of IN/ANY clauses that are badly supported)

* Update Dockerfile -> Go 1.13.7, add build-base (as gcc and friends are needed for SQLite)

* Implement GET endpoints for account_data in clientapi

* Nuke filtering for now...

* Revert "Implement GET endpoints for account_data in clientapi"

This reverts commit 4d80dff4583d278620d9b3ed437e9fcd8d4674ee.

* Implement GET endpoints for account_data in clientapi (#861)

* Implement GET endpoints for account_data in clientapi

* Fix accountDB parameter

* Remove fmt.Println

* Fix insertAccountData SQLite query

* Fix accountDB storage interfaces

* Add empty push rules into account data on account creation (#862)

* Put SaveAccountData into the right function this time

* Not sure if roomserver is better or worse now

* sqlite work

* Allow empty last sent ID for the first event

* sqlite: room creation works

* Support sending messages

* Nuke fmt.println

* Move QueryVariadic etc into common, other device fixes

* Fix some linter issues

* Fix bugs

* Fix some linting errors

* Fix errcheck lint errors

* Make naffka use postgres as fallback, fix couple of compile errors

* What on earth happened to the /rooms/{roomID}/send/{eventType} routing

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-02-13 17:27:33 +00:00
S7evinK
3dfafd4824
Implement missing device management features (#835)
* Implement missing device management features

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add a little more documentation

* Undo changes

* Use non-anonymous struct to decode devices list

* Update sytest-whitelist

* Update sytest-whitelist

* Update sytest-blacklist

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-02-11 12:13:38 +00:00
Behouba Manassé
9937c05bea
Pass cfg by reference around the codebase (#819)
* Pass cfg by reference around the codebase

* Merge branch 'master' into pass-cfg-by-ref

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-02-11 11:18:12 +00:00
Neil Alexander
c20109a573
Implement room version capabilities in CS API (#866)
* Add wiring for querying the roomserver for the default room version

* Try to implement /capabilities for room versions

* Update copyright notices

* Update sytests, add /capabilities endpoint into CS API

* Update sytest-whitelist

* Add GetDefaultRoomVersion

* Fix cases where state package was shadowed

* Fix version formatting

* Update Dockerfile to Go 1.13.6

* oh yes types I remember

* And fix the default too
2020-02-05 18:06:39 +00:00
Neil Alexander
6885c10083
Implement GET endpoints for account_data in clientapi (#861)
* Implement GET endpoints for account_data in clientapi

* Fix accountDB parameter

* Remove fmt.Println
2020-01-29 17:53:05 +00:00
Alex Chen
e239fb10f3
Add missing servers field in /directory/room/:alias response (#732) 2019-10-02 00:09:47 +08:00
Parminder Singh
d21a2fb152 Add auth fallback endpoint (#405)
Also adds support for the recaptcha auth type.
2019-08-15 01:34:49 +08:00
Alex Chen
aa0d22bf50
Implement client single event retrieval (#693) 2019-08-09 17:45:54 +08:00
Alex Chen
324ca22b35
Implement profile retrieval over federation (#726) 2019-08-07 00:02:12 +08:00
Alex Chen
f8d2860765 Replace membership and visibility values with constants (#774)
Signed-off-by: Alex Chen <minecnly@gmail.com>
2019-08-06 15:07:36 +01:00
Sumukha Pk
d283676b9a Implements room tagging. (#694) 2019-08-02 19:17:51 +08:00
Andrew Morgan
33a1392541
Encode URLs properly (#728)
We were escaping the URL before performing any pattern matching on it.
This meant that if you sent data that URLdecoded to a "/", it would count as
a "/" in the URL, potentially causing a 404. This was causing some flaky tests
with some randomly-generated query parameters.

Now, we keep URLs encoded while doing the pattern matching, and only afterwards
do we URL decode each query parameter individually before passing them to their
respective handler functions.

github.com/gorilla/mux was also updated to v1.7.3 to fix a bug with URL encoding and subrouters.
2019-07-03 16:38:50 +01:00
ruben
74827428bd use go module for dependencies (#594) 2019-05-21 21:56:55 +01:00