1/6/14

Frequently Asked Questions - zeromq

FrequentlyAskedQuestions
Fold TableofContents DesignHelp Contributing Support License General PlatformandPatterns Performance Monitoring Security BackwardsCompatibility Building Seealso Who'swatchingthispage?

DesignHelp
I'vearoughideaofhowMQworks,canyouhelpmebuildmyarchitecture? Thisiswhatyoupayexpertsfor,andmanyofusworkingonMQmakeourlivinglikethis.Sowhynotbudgetaweekor twoofexperthelpintoyourprojectplanning,andaskonthelistforsomeonetocomeandhelp?Ifyou'remakinganopen sourceproject,youwillfindthecommunitymoreinterestedingivingtheirtimeforfree. TheotheroptionistoreadtheGuide,whichhassomehundredsofworkedexamplesandisnowavailableasanO'Reilly book.Readit,workthroughtheexamples,andusethemasthebasisforyourfirstexperiments. HowdoImakeMQdoX,Y,orZ? Ifyouaskthisquestionitusuallymeansyou'venotreadtheGuide,orjustskimmeditandnotdigestedit.MQisnota replacementforothermessagingsystems.Itisanewwayofdesigningdistributedcode.Really,theonlywaytodothis profitablyistointernalizehowitworks,whichtakesafewdaysatleast.Whenyou'vedigestedit,you'llseeyourold problemsinanewlight. OK,canyouatleastexplainhowthesocketpatternswork? Eachbindallowspeerstoconnect.Thenumberofconnections(onasingleendpointormultipleendpoints)affectshow thesocketpatternworks.PUSH/DEALERwillrotatemessagesout.PUBdoesmulticast.SUB/PULL/DEALERrotates messagesin.ROUTERrotatesin,andusesaddressedoutput.PAIRalwayssendstoitsuniquepeer,ifany.

Contributing
zeromq.org/area:faq 1/8

1/6/14

Frequently Asked Questions - zeromq

Iwanttosendapatch,howdoIdothat? Seethecontributionpolicy.Wemakethisassimpleasitpossiblycanbe,andaslongasyoufollowtherules,we'llaccept yourpatch,nomatterhowsurprisingitmightbe. What'stheroadmapforMQ? Justtokeepcollectinginterestingproblemsandsolvingthemonebyone.Rightnowlotsofpeoplearethinkingabout security.Perhapstomorrowitwillbesomethingelse.Inanycasethere'snoguidinggeniusbehindMQ,justalotofvery smartpeoplesolvingveryhardproblems. I'vefoundaprobleminMQ,shouldIfileabugreport? Sure,ifyoulike.Buthonestly,there'salargechanceitwilljustsitthereunlessyoufindawaytosendusapatch.Ifyou don'tliketheideaofopeningupthecodeyourself,andthebugiscritical,tryaskingonthezeromqdevmailinglistif anyoneelsewantstohelpyoufixit.Offersofmoneywillalwaysmakethisdiscussioneasier. Whyarethereseveralrepos(zeromq2x,zeromq3x,zeromq4x)? Thesereposarenotformassuse,butforcontributors.Wemovedawayfromgitbranchesbecausetheyarecomplexand fragile.Weuseaseparate"releasefork"formajorversions.Itmaybeunusualbuthasworkedwell.

Support
WheredoIgetpaidsupportforMQ? EmailPieterHintjens,atiMatix,andexplainwhatyouneedintermsofsupport(whatlanguages,platforms,whatkindof servicelevels,whatkindofsupport.He'lleitherputyouintouchwiththerightpeople,orproposeyouapackagetosuityour needs.iMatixbringstogetherlotsoftheexpertswhohelpbuildMQ.

License
CanIuseMQinmyclosedsourceapplication? Yes,MQislicensedundertheLGPLwithastaticlinkingexception.ThislicensegivesyoutheexplicitrighttolinkMQ withclosedsourceapplications(aswellasopensourceapplications). CanIuseMQinmycommercialapplication? Yes.YoumayuseMQinyourcommercialapplicationaslongasyouresubmitanychangesyoumightmaketotheMQ sourcecodeitself.Thelicensehasastaticlinkexceptionthatletsyouredistributethecompiledlibrarywithyour applicationunderyourownterms(nottheLGPL). CanIusepiecesoftheMQcodeinmynonLGPLlibraryorapplication? IfyouincludeanyMQcodeinyourclosedsourceworks,thesebecomederivedworksofMQandmustbelicensed undertheLGPL.TheLGPLdoesmakeanexceptionforheaderfiles. WhydoesMQusetheLGPLandnotamoreliberallicenselikeBSD? Webelieveincommunitybeforesoftware,andinourexperiencetheLGPListherightlicensetobuildthemostsuccessful communityaroundthistechnology. WhataboutpeopleorfirmswhowillnotorcannotusetheLGPL? Theyarefreetochooseanyofthedozensofalternativeproducts,orbuildtheirown. CouldwechangetheMQlicense? Yes,wecouldchangetoalaterversionofLGPL. WhoisthecopyrightownerofMQ?MaybeIcantalktohimorher?

zeromq.org/area:faq

2/8

1/6/14

Frequently Asked Questions - zeromq

MQisownedbyeachcontributor.Therearehundredsofowners.Feelfreetotrackthemdownandaskthemindividually iftheywanttogiveyoutheirworkunderadifferentlicense.

General
WhatversionoflibzmqshouldIuseforanewproject? Allnewdevelopmentshoulduse3.2.Anyexistingprojectsthatrelyon2.2(orearlier)shouldbemigratedastimepermits. While2.2isnotgoingtobeabandoned,the3.2seriesofreleasesisthenewmainlineofdevelopment. HowdoIstarttheMQdaemon? Therearenoservicesordaemonstostart,unlessyoubuildthemyourself.MQisalibrary.Compileandlinkanexample andrunit.MQapplicationsspeaktoeachotherdirectly. HowdoIusearegularBSDsockettocommunicatewithaMQsocket? MQhasitsownwirelevelprotocol.Inorderforaregularsockettocommunicatewithit,thatregularsocketwouldneedto sendmessagesconformingtothatprotocol.Additionally,pleasereadthesectionintheguidedetailinghowMQisnota neutralcarrierforyourdata. CanIuseMQtointeractwithnormalsocketsand,forexample,beabletopingwww.google.comfromaMQ socket? No.SeeprioransweraboutBSDsocketsabove. DoesMQsupportIPmulticast? Yes,MQincludesPragmaticGeneralMulticast(RFC3208)supportusingtheexcellentOpenPGMimplementation. DoesMQstoremessagesondisk? No.Ifyouwanttodothis,youcanbuildyourownstoragequeuesfairlysimply. DoesMQincludeAPIsforserializingdatato/fromthewirerepresentation? No.ThisdesigndecisionadherestotheUNIXphilosophyof"doonethinganddoitwell".InthecaseofMQ,thatone thingismovingmessages,notmarshalingdatato/frombinaryrepresentations. SomemiddlewareproductsdoprovidetheirownserializationAPI.Webelievethatdoingsoleadstobloatedwirelevel specificationslikeCORBA(1055pages).Instead,we'veoptedtousethesimplestwireformatspossiblewhichensure easyinteroperability,efficiencyandreducethecode(andbug)bloat. Ifyouwishtouseaserializationlibrary,thereareplentyofthemoutthere.Seeforexample GoogleProtocolBuffers MessagePack JSONGLib C++BSONLibrary Notethatserializationimplementationsmightnotbeasperformantasyoumightexpect.Youmayneedtobenchmarkyour workloadswithseveralserializationformatsandlibrariesinordertounderstandperformanceandwhich format/implementationisbestforyourusecase(easeofdevelopmentmustalsobeconsidered). I'mtryingtosenddatausingthepgm/epgmtransportandIget"Protocolnotsupported".Why? YouneedtohaveMQbuiltwiththe w i t h p g m optionforthistobeenabled. HowcanIintegrateMQsocketswithnormalsockets?OrwithaGUIeventloop? Youcanusethezmq_poll()functiontopollforeventsonbothMQandnormalsockets.Thezmq_poll()functionaccepts atimeoutsoifyouneedtopollandprocessGUIeventsinthesameapplicationthreadyoucansetatimeoutand periodicallypollforGUIevents.Seealsothereferencedocumentation.

zeromq.org/area:faq

3/8

1/6/14

Frequently Asked Questions - zeromq

Whycan'tIusestandardI/Omultiplexingfunctionssuchasselect()orpoll()onMQsockets? MQsocketisnotastandardPOSIXsocket.Itwouldbegreatifitwas,however,POSIXdoesn'tprovideamechanismto simulatefiledescriptorsinuserspace.ToconvertMQsocketsintoPOSIXfiledescriptorswewouldhaveeithertomove MQtokernelspaceorhackthekerneltoprovidethefunctionalityneeded.Inbothcaseswewouldhavetosacrifice portabilityandsticktoasingleoperatingsystem.Notethatthere'sawaytoretrieveafiledescriptorfromMQsocket (ZMQ_FDsocketoption)thatyoucanpollonfromversion2.1onwards,however,therearesomeseriouscaveatswhen usingit.Checkthedocumentationcarefullybeforeusingthisfeature. Whensendingamultipartmessage,myreceiverdoesn'tgetanymessagepartsuntilthelastpartissent.IsMQ broken? TheMQlibraryguaranteesthatmultipartmessagesaresentandreceivedatomically.Amessagecannotbedeliveredby thelibrarytoyourapplicationuntilallframeshavebeenreceived.Obviously,thiscannothappenwhilethesenderstillhas partspendingorinqueue. Therearetworeasonsformultipartmessagesupportinthelibrary. 1. Thisfunctionsupportsascatter/gatherstyleformessagetransmissionsotheapplicationcanavoidcopyingthe (potentially)largedatatoasinglemessage.Zerocopyisahugeperformancewinparticularlyforlargemessages. 2. Multipartsupportmayhelpaprotocoldesignertologicallyseparatetheirprotocolintoseparateframesatthe applicationlevel.Thismayeaseparsingonthereceivingendsincenotalllanguagessupportfastbittwiddlingfor binarystructureslikeC. DoesMQbuffermyentiremessageinmemorybeforesendingtoitsrecipients? Yes.Soifyouaresendingmultigigabytemessages,itwillconsumeRAMtostorethosemessagesonbothsides.The memoryonthesendersideisfreedafterlastpartofthemessagehasbeensent.Also,seepointabove. IsitpossibletoreceiveEAGAINortoblockwhensendingamultipartmessage? Multipartmessagesaresentasanatomicunit.Ifonepartissentsuccessfully,thenthesocketisguaranteedtonotblock orreturnEAGAINuntilallpartshavebeensent.Ofcourse,ifyourunoutofmemoryinthemeantimethentheguarantee doesn'tmeanmuch,sodon'ttrytosendmessagesthatexceedavailablememory. AreyoutryingtotellmethatMQwon'tmagicallysavemefromoutofmemoryconditions? YouareaperfectcandidateforusingAMQPorJMS.PleaseuseGoogletofindmoreinformationonlibrariesthatsupport thoseplatformsandprotocols. CanIsubscribetomessagesusingregexorwildcards? No.Prefixmatchingonly. WhatportionsoftheMQstructuressurviveacalltofork()? Ifyoumustfork,thentheforkedprocessshouldcreateitsowncontextanditsownsockets.Itisnotsafetoshareacontext orsocketsbetweenaparentanditschild.

PlatformandPatterns
IwanttowriteaprogramusingMQsockets.WhichsockettypesshouldIuse? Pleasereadtheguide.Itcoverseachsocketpatternalongwiththeirusecases.Forfurtherassistance,pleasejointhe mailinglistorIRCandaskforhelp.Pleasedonotbeupsetifthefirstquestionaskedis,"HaveyoureadtheGuideyet?" MymultithreadedprogramkeepscrashinginweirdplacesinsidetheMQlibrary.WhatamIdoingwrong? MQsocketsarenotthreadsafe.ThisiscoveredinsomedetailintheGuide. Theshortversionisthatsocketsshouldnotbesharedbetweenthreads.Werecommendcreatingadedicatedsocketfor eachthread. Forthosesituationswhereadedicatedsocketperthreadisinfeasible,asocketmaybesharedifandonlyifeachthread

zeromq.org/area:faq

4/8

1/6/14

Frequently Asked Questions - zeromq

executesafullmemorybarrierbeforeaccessingthesocket.MostlanguagessupportaMutexorSpinlockwhichwill executethefullmemorybarrieronyourbehalf. Formoreinformation,pleasereadtheGuide. WhydoIseedifferentbehaviorwhenIbindasocketversusconnectasocket? MQcreatesqueuesperunderlyingconnection,e.g.ifyoursocketisconnectedto3peersocketsthereare3messages queues. Withbind,youallowpeerstoconnecttoyou,thusyoudon'tknowhowmanypeerstherewillbeinthefutureandyoucannot createthequeuesinadvance.Instead,queuesarecreatedasindividualpeersconnecttotheboundsocket. Withconnect,MQknowsthatthere'sgoingtobeatleastasinglepeerandthusitcancreateasinglequeueimmediately. ThisappliestoallsockettypesexceptROUTER,wherequeuesareonlycreatedafterthepeerweconnecttohas acknowledgeourconnection. Consequently,whensendingamessagetoboundsocketwithnopeers,oraROUTERwithnoliveconnections,there's noqueuetostorethemessageto. WhenshouldIusebindandwhenconnect? Asaverygeneraladvice:usebindonthemoststablepointsinyourarchitectureandconnectfromthemorevolatile endpoints.Forrequest/replytheserviceprovidermightbepointwhereyoubindandtheclientusesconnect.Likeplainold TCP. Ifyoucan'tfigureoutwhichpartsaremorestable(i.e.peertopeer)thinkaboutastabledeviceinthemiddle,whereboths sidescanconnectto. Thequestionofbindorconnectisoftenoveremphasized.It'sreallyjustamatterofwhattheendpointsdoandiftheylive longornot.Andthisdependsonyourarchitecture.Sobuildyourarchitecturetofityourproblem,nottofitthetool. I'mworriedaboutmyapplicationheartbeatsqueuingupbehindlowerprioritydataandcausingadisconnect.Howcan Isendamessagewithhigherprioritysoitjumpsforwardinthequeue? SeethediscussionofheartbeatingintheGuide. IneedtohavemultiplesocketsshareasingleTCPconnection(host++port).HowcanIaccomplishthis? ThisisbeingaddedtothenextversionoftheZMTPprotocol.Todayyoucanaccomplishthisusingaproxythatsits betweentheexternalTCPaddress,andyourtasks. IsetaHWM(highwatermark)forasocketbutitisn'tworking! That'snotaquestion.Also,pleasebecertaintoreadthemanpageforzmq_setsockopt()closely.Certainsocketoptions onlytakeeffectforsubsequentzmq_bind/zmq_connectcalls.Werecommendsettingallsocketoptionsbeforemakingany callstozmq_bind/zmq_connectthatwaywedon'thavetorememberthesetinyimplementationdetailsandcanfocuson writinggreatcode. Also,readthenextquestionaboutHWMbelow. HowdoestheHWM(highwatermark)workwithanysockettype? Itworksthefollowingwayrightnow: TheI/Othreadreadsmessagesfromthepipeandpushesthemtothenetwork.Ifnetworkisnotabletoacceptmoredata (e.g.TCPbackpressureisapplied)itstopsreadingmessagesfromthepipeandwaitsuntilthenetworkisreadyfor acceptingmoredata. Intheapplicationthread,messagesaresimplypushedtothepipewhenzmq_send()iscalled.Ifthepipeisfull(HWMis reached)themessageisdropped. Theproblemwiththeaboveapproachisthatwhenyousendalotofmessagesisaquicksequence(e.g.sendingsmall messagesinatightloop)themessagesarestoredinthepipeuntilitisfullandthesubsequentmessagesaresimply dropped.Thesenderisnotevennotifiedaboutthefactthatmessagesaredisappearing.

zeromq.org/area:faq

5/8

1/6/14

Frequently Asked Questions - zeromq

Themaincoredeveloperishopefulthatsomecommunitymemberswillvolunteertoassistinreplacingthismechanism witharateflowcontrolmechanism. HowcanIflushallmessagesthatareintheMQsocketqueue? Thereisnoexplicitcommandforflushingaspecificmessageorallmessagesfromthemessagequeue.Youmayset ZMQ_LINGERto0andclosethesockettodiscardanyunsentmessages. Whenrunning"makecheck"onOSX,howdoIfixthefailures? Pleaserefertothetuningguideforplatformspecifictuning.Inshort,OSXhassomelowdefaultsthatthetestsoverrunso theyneedtobetuned.

Performance
WhatistheoptimalnumberofI/Othreadsforbestperformance? Thebasicheuristicistoallocate1I/Othreadinthecontextforeverygigabitpersecondofdatathatwillbesentand received(aggregated).Further,thenumberofI/Othreadsshouldnotexceed(number_of_cpu_cores1). ThegraphinthetestresultsshowsthatMQisslowerthanTCP/IP.What'sthepointthen? Obviously,youwouldexpectsystemworkingontopofTCPtohavehigherlatenciesthanTCP.Anythingelsewouldbe simplyspeakingsupernatural.However,throughputisadifferentmatter.MQgetsyoumorethroughputthanTCPhas usingintelligentbatchingalgorithms.MoreoverMQdeliversvalueaddovertheTCP.Asynchronicity,messagequeueing, routingbasedonbusinesslogic,multicastetc. HowcomeMQhashigherthroughputthanTCPalthoughit'sbuiltontopofTCP? Avoidingredundantnetworkingstacktraversalscanimprovethroughputsignificantly.Inotherwords,sendingtwo messagesdownthenetworkingstackinonegotakesmuchlesstimethensendingeachofthemseparately.This techniqueisknownasmessagebatching. Whensendingmessagesinbatchesyouhavetowaitforthelastonetosendthewholebatch.Thiswouldmakethe latencyofthefirstmessageinthebatchmuchworse,wouldn'tit? MQbatchesmessagesinopportunisticmanner.Ratherthanwaitingforapredefinednumberofmessagesand/or predefinedtimeinterval,itsendsallthemessagesavailableatthemomentinonego.Imaginethenetworkinterfacecard isbusysendingdata.OnceitisreadytosendmoredataitasksMQfornewmessages.MQsendsallthemessages availableatthemoment.Doesitharmthelatencyofthefirstmessageinthebatch?No.Themessagewon'tbesent earlieranywaybecausethenetworkingcardwasbusy.Onthecontrary,latencyofsubsequentmessageswillbeimproved becausesendingsinglebatchtothecardisfasterthensendinglotofsmallmessages.Ontheotherhand,ifnetworkcard isn'tbusy,themessageissentstraightawaywithoutwaitingforfollowingmessages.Thusit'llhavethebestpossible latency. MQ'slatencyisnice,butisthereawaytomakeitevenlower? WeareworkingondeliveringMQoveralternativenetworkingstacks,thushavingadvantageoffeatureslikekernel bypass,avoidingTCP/IPoverhead,usinghighperformancenetworkinghardwareetc.Thatwaywecangetthelatencyas lowas10microseconds. WhyamIonlygetting40MbpsperformancewhensendingmessagesusingPGM? PGMusesratelimitingonthesenderside.Bydefaultthislimitissetto40Mbps.YoucansetitusingtheZ M Q _ R A T E option tozmq_setsockopt(). DoestheMQlibrarydisabletheNaglealgorithm(TCPNODELAY)? Yes.

Monitoring

zeromq.org/area:faq

6/8

1/6/14

Frequently Asked Questions - zeromq

HowdoIdeterminehowmanymessagesareinqueue? Thisisn'tpossible.AtanygiventimeamessagemaybeintheMQsenderqueue,thesender'skernelbuffer,onthewire, inthereceiver'skernelbufferorinthereceiver'sMQreceiverqueue.Furthermore,aMQsocketcanbindand/orconnect tomanypeers.Eachpeermayhavedifferentperformancecharacteristicsandthereforeadifferentqueuedepth.Any "queuedepth"numberisalmostcertainlywrong,soratherthanprovideincorrectinformationthelibraryavoidsproviding anyviewintothisdata. HowcanIbenotifiedthatapeerhasconnected/disconnectedfrommysocket? MQsocketscanbindand/orconnecttomultiplepeerssimultaneously.Thesocketsalsotransparentlyprovide asynchronousconnectionandreconnectionfacilities.Atthistime,noneofthesocketswillprovidenotificationofpeer connect/disconnect.Thisfeatureisbeinginvestigatedforafuturerelease. HowcanIretrievealistofallconnectedpeers? Thisisnotsupported. HowcanIautodiscoverservicesprovidedbyaMQbasedapplication? Thistypeoffacilityisnotsupportedbythelibrary.SuchatoolcouldbebuiltontopofMQ.

Security
IsittruethatitisnotsafetouseMQovertheinternetbecauseitwillcrash? EarlierversionsoftheMQlibrary(before2.1)werenotveryresilientagainst"fuzzing"attacks.Amalformedpacketor garbagedatacouldcauseanoldversionofthelibrarytoassertandexit.Sincethereleaseof2.1,allreportedcasesof assertionscausedbybaddatahavebeenfixed.Ifyourtestinguncoversaprobleminthisarea,pleasefileabugreport. WhatsecurityfeaturesdoesMQsupport? Noneatthemomentbutthisisbeingaddedtothenextversionoftheprotocol.PeoplehavesuccessfullybuiltDTLS, CurveCP,andothersecurityprotocolsoverMQ. IreadsomewherethatIhavetorunmyapplicationasrootifIwanttousePGM,isthistrue? Theepgm://transportusesPGMencapsulatedinUDPpacketsanddoesnotrequireanyspecialpermissions. Ifyouneedtousetherawpgm://transportthenyourapplicationmustbeabletocreaterawsockets,whichmeanseither runningasrootorwithcapabilitiestodoso.OnamodernLinuxdistributionwithcapabilitiesenabledyoucanusethe followingtorunanapplicationwiththeC A P _ N E T _ R A W capabilityenabled:
$ s u d o e x e c c a p ' c a p _ n e t _ r a w = e p ' p g m s e n d m o o

FormoredetailspleaseseetherelevantOpenPGMwikipage.

BackwardsCompatibility
0MQ/3.2.2stableandlaterreleasesarecompatiblewith2.2and2.1.Allother3.xversionsupto3.2.1areonlycompatible withthemselves,whichisunfortunatebutcausedbyreleases3.0and3.1whichbroketheprotocolwithoutaddingany versioninformation.Oneofthesymptomsofusingtheseincompatiblereleases(3.0.x,3.1.x,3.2.0,3.2.1)withstable releasesisthatrequestreplywillworkinonedirectiononly.

Building
WhatpackagesdoIneedtobuildMQonUbuntu?

zeromq.org/area:faq

7/8

1/6/14

Frequently Asked Questions - zeromq

Youneedb u i l d e s s e n t i a l and,forpreMQ/3.0versions,u u i d d e v .Sos u d o a p t g e t i n s t a l l b u i l d e s s e n t i a l u u i d d e v .Youcanalsoruns u d o a p t g e t b u i l d d e p l i b z m q 0 whichinstallsallthebuilddependenciesofthe libzmq0ubuntupackage. Aftercloningthegithubrepository,Ican'tbuildthelibrarybecausethe'configure'scriptdoesn'texist!WhatdoIdo? YouneedautotoolsinstalledforyourOSsothattheconfigurescriptcanbegenerated.Runthiscodetogeneratethat script.

$ . / a u t o g e n . s h

Websitedesignandc ontentisc opyright(c )2013iMatixCorporation.Contac tusforprofessionalsupport. Sitec ontentlic ensedunderc c bysa3.0MQisc opyright(c )Copyright(c )20072013iMatix CorporationandContributors.MQisfreesoftwarelic ensedundertheLGPL.MQ,ZeroMQ,0MQ,and "T heIntelligentT ransportLayer"aretrademarksofiMatixCorporation.T ermsofUsePrivac yPolic y

zeromq.org/area:faq

8/8