Dgraph的Key解密
[if !supportLists]一. [endif]Dgrap的key有預(yù)定義有幾種:
//key類型
?????? ByteData???? = byte(0x00)
?????? ByteIndex??? = byte(0x02)
?????? ByteReverse? = byte(0x04)
?????? ByteCount??? = byte(0x08)
?????? ByteCountRev= ByteCount | ByteReverse
?????? //前綴
?????? defaultPrefix= byte(0x00)
?????? byteSchema??? = byte(0x01)
?????? byteType????? = byte(0x02)
[if !supportLists]1.??? [endif]DataKey:
buf := make([]byte, 2+len(attr)+2+8)??????????????? ????????????????????
00411097109101000000040
defaultPrefix Length? ?? n ????a??? m? ?????? ?e? ByteData?? Uid:1024從QL層的謂語值sid計算
uid的轉(zhuǎn)換函數(shù):
func (bigEndian) PutUint64(b []byte, vuint64) {
?????? _= b[7] // early bounds check to guarantee safety of writes below
?????? b[0]= byte(v >> 56)
?????? b[1]= byte(v >> 48)
?????? b[2]= byte(v >> 40)
?????? b[3]= byte(v >> 32)
?????? b[4]= byte(v >> 24)
?????? b[5]= byte(v >> 16)
?????? b[6]= byte(v >> 8)
?????? b[7]= byte(v)
}
[if !supportLists]2.??? [endif]SchemaKey:
buf := make([]byte, 1+2+len(attr))
10397103101
byteSchema?Length Length? a??????? ?g???????? e???
[if !supportLists]3.??? [endif]Typekey:
buf := make([]byte, 1+2+len(typeName))
103737884
?byteType ?Length Length? typeName:INT
[if !supportLists]4.??? [endif]IndexKey:
buf := make([]byte,2+len(attr)+2+len(term))??????????????????????????? ? term---->
00??? 4110971091012??? 16510810599101
defaultPrefix length length? attr:n?? a????? m?? e??? ByteIndex?Identifier??A???? l????i?? ???c?? ??????e
identifier:
const (
?????? IdentNone???? = 0x0
?????? IdentTerm???? = 0x1
?????? IdentExact??? = 0x2
?????? IdentYear???? = 0x4
?????? IdentMonth??? = 0x41
?????? IdentDay????? = 0x42
?????? IdentHour???? = 0x43
?????? IdentGeo????? = 0x5
?????? IdentInt????? = 0x6
?????? IdentFloat??? = 0x7
?????? IdentFullText= 0x8
?????? IdentBool???? = 0x9
?????? IdentTrigram? = 0xA
?????? IdentHash???? = 0xB
?????? IdentCustom?? = 0x80
)
[if !supportLists]5.??? [endif]ReverseKey
buf := make([]byte, 2+len(attr)+2+8)
00??? 4110971091014????????
00000040
defaultPrefix length length? attr:n?? a????? m?? e??? ByteReverse??uid:1024
[if !supportLists]6.??? [endif]CountKey
buf := make([]byte, 1+2+len(attr)+1+4)
00??? 4110971091014/8???????????? 003230
defaultPrefix length length? attr:n?? a????? m?? e?ByteCount/ByteCountRev??count:998
count的轉(zhuǎn)換:
func (bigEndian) PutUint32(b []byte, vuint32) {
?????? _= b[3] // early bounds check to guarantee safety of writes below
?????? b[0]= byte(v >> 24)
?????? b[1]= byte(v >> 16)
?????? b[2]= byte(v >> 8)
?????? b[3]= byte(v)
}
[if !supportLists]二.???[endif]Key的使用-寫入data
[if !supportLists]1.??? [endif]最初Dgraph會把GraphQL解析的NQuad對象請求轉(zhuǎn)換成DirectedEdge對象
type NQuad struct {? ?????????????????????????????????? ?????? // GraphQL中RDF NQuad數(shù)據(jù)格式
?????? Subject????????????? string?
?????? Predicate??????????? string?
?????? ObjectId???????????? string??
?????? ObjectValue????????? *Value?
?????? Label??????????????? string
?????? Lang???????????????? string?
?????? Facets?????????????? []*Facet
?????? XXX_NoUnkeyedLiteralstruct{}
?????? XXX_unrecognized???? []byte??
?????? XXX_sizecache??????? int32?
}
type DirectedEdge struct {??????????????????????????????????? //Represents the original uid -> value edge
?????? Entity?????????????? uint64????????? ?????????? //實(shí)體 RDF里的subject,uid
?????? Attr???????????????? string???????? ??????????? //屬性—謂語 predicate
?????? Value??????????????? []byte??????? ???????????? //值object
?????? ValueType??????????? Posting_ValType?? ?????? //值類型 包括binaryint? bool uid pasword類型
?????? ValueId????????????? uint64???????? ?????????? //值的id
?????? Label??????????????? string????????? ????????? //label
?????? Lang???????????????? string???????? ?????????? //語言
?????? Op?????????????????? DirectedEdge_Op //操作setor delete,默認(rèn)是set
?????? Facets?????????????? []*api.Facet??? ???????????? //predicate上的面
?????? XXX_NoUnkeyedLiteralstruct{}
?????? XXX_unrecognized???? []byte
?????? XXX_sizecache??????? int32
}
type Posting struct {??????????????? //Protocol Buffers協(xié)議,提供序列化數(shù)據(jù)結(jié)構(gòu),用于raft協(xié)議通信和存儲,取出數(shù)據(jù)
?????? Uid???????? uint64????????????
?????? Value?????? []byte????????????
?????? ValType???? Posting_ValType??? //值類型 包括binaryint? bool uid pasword類型
?????? PostingTypePosting_PostingType //Posting_REF uid類型,Posting_VALUE值類型,Posting_VALUE_LANG語言類
?????? LangTag???? []byte?????????????
?????? Label?????? string?????????????
?????? Facets????? []*api.Facet???????
?????? Op?????????????????? uint32??
?????? StartTs????????????? uint64??
?????? CommitTs???????????? uint64??
?????? XXX_NoUnkeyedLiteralstruct{}
?????? XXX_unrecognized???? []byte?
?????? XXX_sizecache??????? int32?
}
============================================================================
[if !supportLists]2.??? [endif]驗證并轉(zhuǎn)換DirectedEdge對象的ValueType,Value列轉(zhuǎn)換成Posting對象
執(zhí)行mutation時,wokerserver執(zhí)行worker/mutation.go下的runMutation函數(shù)
// runMutation goes through all the edgesand applies them.
在其中,ValidateAndConvert驗證DirectedEdge對象的TypeID和Posting對象的Posting_ValType是否一致,并嘗試把DirectedEdge對象的Value列的值轉(zhuǎn)換成Posting_ValType類型,再轉(zhuǎn)換成[]byte,驗證是否成功
//驗證的過程中Convert和Marshal的轉(zhuǎn)換有重復(fù)
如果都成功,修改DirectedEdge對象的ValueType的值,Value改為剛才轉(zhuǎn)換的[]byte
============================================================================
[if !supportLists]3.??? [endif]使用DataKey方法產(chǎn)生key,并在提交前使用Key來檢查本地緩存是否有數(shù)據(jù)
加工的edge提交的Attr和Enity使用DataKey方法產(chǎn)生key
再根據(jù)key從事務(wù)中本地緩存中找到對應(yīng)的Posting對象(四層的truct嵌套為List對象),如果緩存為空,從badgerdb中查詢.
[if !supportLists]4.??? [endif]使用Posting對象提交事務(wù)并處理索引
使用List對象的AddMutationWithIndex方法來提交事務(wù),如果謂語有索引,通過檢查postlist是否有提交語句中的值的uid(通過hash64值獲得),并檢查當(dāng)前提交與新提交的post中的 值和值的類型是否相同.
最后使用list.addMutation來應(yīng)用mutation,通過檢查是否是upsert屬性,如果是,檢查key沖突,如果不是只檢查數(shù)據(jù)沖突
使用DirectedEdge的方法NewPosting,轉(zhuǎn)換一個Posting對象:
t *pb.DirectedEdge
?????? return&pb.Posting{
????????????? Uid:???????? t.ValueId,
????????????? Value:?????? t.Value,
????????????? ValType:???? t.ValueType,
????????????? PostingType:postingType,
????????????? LangTag:???? []byte(t.Lang),
????????????? Label:?????? t.Label,
????????????? Op:????????? op,
????????????? Facets:????? t.Facets,
?????? }
[if !supportLists]5.??? [endif]處理提交
最終這個Posting對象放到ProtocolBuffs中PostingList對象中的mutationMap這個map,用Posting的StartTs值作為下標(biāo);增加,并把key放到txn的deltas中確保順序
[if !supportLists]①???? [endif].當(dāng)事務(wù)提交到磁盤的時候使用事務(wù)的CommitToDisk方法,調(diào)用Plist的GetMutation方法:
[if !supportLists]1.???[endif]先從deltas中獲取key并添加到keys切片中
[if !supportLists]2.???[endif]按照順序從keys取出key,再根據(jù)key從緩存中獲取List,從而取出mutationMap中的Plist,進(jìn)過Marsh(),這里取出的是value值
[if !supportLists]3.???[endif]使用key和value值組成entry寫入數(shù)據(jù),調(diào)用badgerdb的sendToWriteCh方法寫入Entry,
這里的key實(shí)際把key在末尾增加八個byte并把MaxUint64減去CommitTs的值寫入后八位byte
[if !supportLists]②???? [endif].?如果是提交到內(nèi)存中, rebuild index的時候使用
第一步一樣,但是不需要從緩存取出數(shù)據(jù),而是直接使用list.CommitMutation方法提交,只需要傳參txn.StartTs和本地提交的commitTs,但是只處理事務(wù)的PostingList的CommitTs值.再使用List.MarshalToKv獲取key value并使用badger的TxnWriter.SetAt方法寫入key value值,寫入預(yù)寫日志之后Flush
寫入內(nèi)存如果發(fā)生沖突5ms重試,寫入磁盤10ms重試
[if !supportLists]三.? [endif]其他key的使用
[if !supportLists]1.??? [endif]SchemaKey:
當(dāng)發(fā)生schema改變或者predicate遷移組時使用txn.SetWithMeta方法把變更寫入txn.writes和pendingWrites,最后使用txn.CommitWith或者txn.Commit提交
[if !supportLists]2.??? [endif]TypeKey:
在predicate的類型變更時使用,調(diào)用的方法和SchemaKey一樣
[if !supportLists]3.??? [endif]IndexKey
通過Key獲取list對象, key作為迭代器的PreFix
[if !supportLists]4.??? [endif]ReverseKey
[if !supportLists]5.??? [endif]CountKey
?
?
最終保存在badger中的數(shù)據(jù)的結(jié)構(gòu):
?
//Entry provides Key, Value, UserMeta and ExpiresAt. This struct can be used bythe user to set data.
typeEntry struct {
???? Key??????[]byte
???? Value????[]byte
???? UserMeta?byte
???? ExpiresAt uint64 // time.Unix
???? meta?????byte
?
???? // Fields maintained internally.
???? offset uint32
}
