Commit Graph

1544 Commits

Author SHA1 Message Date
Neil Alexander
b4c3692dcc
Optimise CheckServerAllowedToSeeEvent (#1602)
* Try to limit how many state events we have to unmarshal

* Comments
2020-12-02 11:45:50 +00:00
Ariadne Conill
d5b8260196
syncapi/requestpool: fix initial sync logic error in appendAccountData() (#1594)
* requestpool: fix initial sync logic error in appendAccountData()

In initial sync, req.since is no longer nil, but instead,
req.since.PDUPosition() and req.since.EDUPosition() returns 0.

This ensures forgotten rooms do not come back as zombies.

* syncapi/requestpool: reintroduce req.since == nil check
2020-12-01 17:01:34 +00:00
bodqhrohro
c667a19775
Assign CREATEDB permission to the test user (#1591)
During the tests databases get recreated, and this fails despite of the
user being the owner of a dropped database. Maybe related to certain
PostgreSQL version.

Signed-off-by: Bohdan Horbeshko <bodqhrohro@gmail.com>
2020-12-01 16:59:47 +00:00
Neil Alexander
c16abb089c
Give fsAPI to keyserver in polylith/gobind 2020-12-01 10:52:02 +00:00
Neil Alexander
9c52f82736
Version 0.3.1 2020-11-20 11:38:58 +00:00
Neil Alexander
c636be5070
Update last seen on sync requests (#1593)
* Update last seen on sync requests

* Fix MSC2836 unit tests

* Only update once per minute

* Remove debug logging

* Configurable option

* Simplify updateLastSeen/cleanLastSeen
2020-11-20 11:29:02 +00:00
alexkursell
13cbd50dc2
Add last_seen_ip and last_seen_ts to /devices response (#1592) 2020-11-20 09:26:50 +00:00
Kegsay
6353b0b7e4
MSC2836: Threading - part one (#1589)
* Add mscs/hooks package, begin work for msc2836

* Flesh out hooks and add SQL schema

* Begin implementing core msc2836 logic

* Add test harness

* Linting

* Implement visibility checks; stub out APIs for tests

* Flesh out testing

* Flesh out walkThread a bit

* Persist the origin_server_ts as well

* Edges table instead of relationships

* Add nodes table for event metadata

* LEFT JOIN to extract origin_server_ts for children

* Add graph walking structs

* Implement walking algorithm

* Add more graph walking tests

* Add auto_join for local rooms

* Fix create table syntax on postgres

* Add relationship_room_id|servers to the unsigned section of events

* Persist the parent room_id/servers in edge metadata

Other events cannot assert the true room_id/servers for the
parent event, only make claims to them, hence why this is
edge metadata.

* guts to pass through room_id/servers

* Refactor msc2836 to allow handling from federation

* Add JoinedVia to PerformJoin responses

* Fix tests; review comments
2020-11-19 11:34:59 +00:00
bodqhrohro
1cf9f20d2f
Allow = in user identifiers (#1590)
While I was breaking through all the TDD bureaucracy to finally push this fix,
it turned out that it already got fixed in #1578. Still I push the rest of the
changes (basically, everything except of actually fixing the bug ;)).

`=` is crucial for matrix-bifrost which turns XMPP @'s into =40.

Signed-off-by: Bohdan Horbeshko <bodqhrohro@gmail.com>
2020-11-18 22:56:06 +00:00
Neil Alexander
e65a3443a4
Fix paths in quickstart 2020-11-18 15:11:48 +00:00
Neil Alexander
265cf5e835
Protect txnReq.newEvents with mutex (#1587)
* Protect txnReq.newEvents and txnReq.haveEvents with mutex

* Missing defer

* Remove t.haveEventsMutex
2020-11-18 11:31:58 +00:00
TR_SLimey
3bcb003248
Add IPv6 setting to nginx conf (#1582)
* Added IPv6 setting to nginx conf

https://matrix.to/#/!yomrOFwgFXzmeMAbzX:matrix.org/$sIkSxL002BKottY1Fop-bkusuyCJ1Y0zIFzqSWDYRrc?via=matrix.org&via=privacytools.io&via=dendrite.link

* Add IPv6 support to polylith nginx conf also
2020-11-18 10:33:17 +00:00
David Spenler
35ea55e70b
Implemented whois endpoint (#1573)
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-17 10:07:03 +00:00
Aaron Raimist
d3b3371856
Update README to remove mention that read receipts are not implemented (#1586)
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2020-11-17 08:43:22 +00:00
Neil Alexander
20a01bceb2
Pass pointers to events — reloaded (#1583)
* Pass events as pointers

* Fix lint errors

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update to matrix-org/gomatrixserverlib#240
2020-11-16 15:44:53 +00:00
Neil Alexander
d8b526b603
Update docker readme 2020-11-16 11:27:59 +00:00
Neil Alexander
bfa9df4a5a
Acknowledgements in changelog 2020-11-16 11:25:24 +00:00
Neil Alexander
3856e547de
Version 0.3.0 2020-11-16 11:13:05 +00:00
Neil Alexander
3034d8e805
Update sytest lists to blacklist flaky 'Forgotten room messages cannot be paginated' test 2020-11-16 10:51:55 +00:00
Mayeul Cantan
af41fcadc4
Fix Dendrite not backfilling on world_readable rooms (#1575)
The previous implementation was only checking if room history was
"shared", which it wasn't for rooms where a user was invited, or world
readable rooms.
This implementation leverages the IsServerAllowed method, which already
implements the complete verification algorithm.

Signed-off-by: `Mayeul Cantan <oss+matrix@mayeul.net>`

Co-authored-by: Kegsay <kegan@matrix.org>
2020-11-16 10:47:16 +00:00
Felix Hanley
41c0b5acfb
Update build script to run on any POSIX (#1577)
Signed-off-by: Felix Hanley <felix@userspace.com.au>
2020-11-16 10:09:53 +00:00
S7evinK
cb41c992f7
Add -version flag (#1581)
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
2020-11-16 09:54:58 +00:00
Neil Alexander
b7e288084c
Update README.md 2020-11-15 12:39:11 +00:00
Neil Alexander
8ce740d949
Fix username regex to allow = character (#1578) 2020-11-12 10:36:54 +00:00
S7evinK
bcb89ada5e
Implement read receipts (#1528)
* fix conversion from int to string yields a string of one rune, not a string of digits

* Add receipts table to syncapi

* Use StreamingToken as the since value

* Add required method to testEDUProducer

* Make receipt json creation "easier" to read

* Add receipts api to the eduserver

* Add receipts endpoint

* Add eduserver kafka consumer

* Add missing kafka config

* Add passing tests to whitelist

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Fix copy & paste error

* Fix column count error

* Make outbound federation receipts pass

* Make "Inbound federation rejects receipts from wrong remote" pass

* Don't use errors package

* - Add TODO for batching requests
- Rename variable

* Return a better error message

* - Use OutputReceiptEvent instead of InputReceiptEvent as result
- Don't use the errors package for errors
- Defer CloseAndLogIfError to close rows
- Fix Copyright

* Better creation/usage of JoinResponse

* Query all joined rooms instead of just one

* Update gomatrixserverlib

* Add sqlite3 migration

* Add postgres migration

* Ensure required sequence exists before running migrations

* Clarification on comment

* - Fix a bug when creating client receipts
- Use concrete types instead of interface{}

* Remove dead code
Use key for timestamp

* Fix postgres query...

* Remove single purpose struct

* Use key/value directly

* Only apply receipts on initial sync or if edu positions differ,
otherwise we'll be sending the same receipts over and over again.

* Actually update the id, so it is correctly send in syncs

* Set receipt on request to /read_markers

* Fix issue with receipts getting overwritten

* Use fmt.Errorf instead of pkg/errors

* Revert "Add postgres migration"

This reverts commit 722fe5a04628882b787d096942459961db159b06.

* Revert "Add sqlite3 migration"

This reverts commit d113b03f6495a4b8f8bcf158a3d00b510b4240cc.

* Fix selectRoomReceipts query

* Make golangci-lint happy

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-09 18:46:11 +00:00
S7evinK
eccd0d2c1b
Implement forgetting about rooms (#1572)
* Add basic storage methods

* Add internal api handler

* Add check for forgotten room

* Add /rooms/{roomID}/forget endpoint

* Add missing rsAPI method

* Remove unused parameters

* Add passing tests

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add missing file

* Add postgres migration

* Add sqlite migration

* Use Forgetter to forget room

* Remove empty line

* Update HTTP status codes

It looks like the spec calls for these to be 400, rather than 403: https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-rooms-roomid-forget

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-05 10:19:23 +00:00
Neil Alexander
2ce2112ddb
Update gjson for safe integer ranges 2020-11-05 09:42:28 +00:00
Petter Jakub Økland
fe339c8b98
Created polylith reverse proxy sample for Hiawatha (#1571) 2020-10-28 16:37:56 +00:00
S7evinK
d5675feb96
Add possibilty to configure MaxMessageBytes for sarama (#1563)
* Add configuration for max_message_bytes for sarama

* Log all errors when sending multiple messages

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add missing config

* - Better comments on what MaxMessageBytes is used for
- Also sets the size the consumer may use
2020-10-27 14:11:37 +00:00
Neil Alexander
c5888bb64c
Update Docker sample configs 2020-10-27 13:58:58 +00:00
Neil Alexander
0af35bec1a
Update documentation (#1569)
* Update dendrite-config.yaml

* Update README.md

* Update INSTALL.md

* Update INSTALL.md
2020-10-26 22:09:13 +00:00
Neil Alexander
59428cdde3
Version 0.2.1 (#1558)
* Version 0.2.1

* Credit where credit due

* Update CHANGES.md

* Include PRs
2020-10-22 12:38:12 +01:00
Adit Sachde
b66b2eab32
Document dendrite server key format (#1520)
Co-authored-by: Kegsay <kegan@matrix.org>
2020-10-22 12:21:31 +01:00
Neil Alexander
f32320a235
Exclude old state events from sync when consumed from roomserver (#1548)
* Exclude old room events from sync when consumed from roomserver

* Fix comment

* Experimental hack
2020-10-22 11:50:48 +01:00
Neil Alexander
3afc623098
Fix RewritesState bug (#1557)
* Set RewritesState once

* Check if any new state provided

* Obey rewritesState

* Don't nuke everything the sync API knows when purging state

* Fix panic from duplicate insert

* Consistency

* Use HasState

* Remove nolint

* Clean up joined rooms on state rewrite
2020-10-22 10:39:16 +01:00
Neil Alexander
04dc019e5e
Don't set empty state snapshots 2020-10-21 16:21:36 +01:00
Neil Alexander
534f9a9eb6
Refactor forward extremities (#1556)
* Add resolve-state helper

* Tweaks

* Refactor forward extremities, again

* Tweaks

* Minor optimisation

* Make path a bit clearer

* Only process state/membership if forward extremities have changed

* Usage comments in resolve-state
2020-10-21 15:37:07 +01:00
Vladislav Povalyaev
e4f3f38f35
Update development percentages (#1553) 2020-10-20 20:49:03 +01:00
Kegsay
6b8791b868
Always call overridden form of GetLatestEventsForUpdate (#1554)
This ensures we don't make txns on sqlite still, which can cause
'database is locked' errors.
2020-10-20 19:32:33 +01:00
Neil Alexander
7612f64e3c
Update gomatrixserverlib to matrix-org/gomatrixserverlib#220 2020-10-20 17:23:04 +01:00
Neil Alexander
24e38c4135
Internal HTTP APIs over H2C (#1541)
* H2C on internal HTTP because SCIENCE

* Update comments
2020-10-20 17:13:12 +01:00
Neil Alexander
7ca89ef511
Update gomatrixserverlib 2020-10-20 17:10:37 +01:00
Neil Alexander
6a16d46fba
Version 0.2.0 (#1551)
* v0.2.0-rc2

* Update CHANGES.md (also with some markdown lint suggestions)

* Update version number

* Update CHANGES.md

* Update CHANGES.md

* Update CHANGES.md

* Add known issue

* Update CHANGES.md
2020-10-20 16:46:53 +01:00
Neil Alexander
39c7a8915c
Multi-personality polylith binary (#1552)
* Initial work oon multipersonality binary

* Remove old binaries

* Monolith and polylith binaries

* Better logging

* dendrite-poly-multi

* Fix path

* Copyright notices etc

* Tweaks

* Update Docker, INSTALL.md

* Take first argument if flags package doesn't find any args

* Postgres 9.6 or later, fix some more Docker stuff

* Don't create unnecessary e2ekey DB

* Run go mod tidy
2020-10-20 16:11:24 +01:00
Neil Alexander
6c3c621de0
Remove invalid state delta check (#1550) 2020-10-20 12:36:16 +01:00
Kegan Dougal
837c295c26 Linting 2020-10-20 12:29:53 +01:00
Pika
53a745f333
fix create-account (#1546) 2020-10-20 11:47:37 +01:00
Kegsay
eb86e2b336
Fix sqlite locking bugs present on sytest (#1543)
* Fix sqite locking bugs present on sytest

Comments do the explaining.

* Fix deadlock in sqlite mode

Caused by starting a writer whilst within a writer

* Only complain about invalid state deltas for non-overwrite events

* Do not re-process outlier unnecessarily
2020-10-20 11:42:54 +01:00
Neil Alexander
92982a402f
Update Docker (#1542)
* Separate Docker images, rather than tags

* Allow specifying tag to build/push/pull

* Include goose in Docker builds
2020-10-20 11:34:22 +01:00
Neil Alexander
a71360d099
Update INSTALL.md (#1549) 2020-10-20 11:22:46 +01:00