|Page Name:||Application Example: Distributed Search Engine|
The idea in this chapter has not been tested in practice yet. It's just a thought, a starting point for an experiment.
The Gist of the Idea
A document or part of a document resides at a Silktorrent packet. A search query calculates a rational number, a distance, between the query and the Silktorrent packet. The question to the network of nodes is: "Do You have a Silktorrent packet that has a MAXIMUM distance of L from the point that is determined by a search-engine specific, pre-defined, function with the following input parameters, param_1,...,param_N?" Some of the nodes on the network will lie, but the only possible, specified, answers to that question are:
- I do not have it, nor was I able to reach anybody within N hops, who has it.
- I have it, the Silktorrent packet name is foo and the distance is bar.
- I do not have it, but ask from node N_x1. Either the N_x1 has it or it can refer to some other node that either has it itself or can give a new node name. The Silktorrent packet name is foo and the distance is bar.
Each node maintains its own, personal, table of node reputations. The query initiator asks the packet owner directly for the packet and verifies that the promised distance between the query and the packet match. In terms of search hops the nodes closer to the liar get higher penalty points than the nodes further from the liar. The rest is a matter of custom heuristics that can be very creative. The liars might lie only occasionally, at random moments or according to some heuristic or statistical distribution, according to instructions from censors, etc. The search engine might be designed to search for text, but the algorithm works also on other data types, for example, 3D shapes, maps, images, sounds, etc.
Partial list of Sources of Inspiration
- Cubit (archival copy,, local_copy, source)
- Cooperative Domain Name System, CoDoNS (archival copy)
- YaCy P2P search engine
- Gigablast search engine
Some Related References
- Princeton CASS: Content-Aware
- Princeton Shape Retrieval
and Analysis Group (archival copy,local