U ¿d=I@sRdZddlmZddlmZddlmZddlmZddlmZddl m Z ddl m Z dd l mZdd l mZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddZddZddZddZddZddZddZd d!Z d"d#Z!d$d%Z"d&d'Z#d6d)d*Z$d+d,Z%d-d.Z&d/d0Z'd1d2Z(d3d4Z)d5S)7z> Appraise evaluation framework See LICENSE for usage details ) defaultdict) OrderedDict)md5)JSONDecodeError)load)User) CommandError)Campaign) CampaignTeam)LANGUAGE_CODES_AND_NAMES)validate_language_code)CAMPAIGN_TASK_TYPES)Market)Metadata)ObjectID) TaskAgendac Cs||}|dks||dkr dS||}g}d}t|D]F}g}t|D]} || |} || | }qH|d7}|t|q8|S)zK Creates task maps, uniformly distributed across given annotators. rN)rangeappendtuple) annotatorstasks redudancyZ _total_tasksZ_tasks_per_annotator_resultsZ_current_task_idZ_unused_annotator_idZ_annotator_tasksZannotator_tasktask_idr=/var/www/rival/public_html/translation-eval/Campaign/utils.py_create_uniform_task_maps    rcCs"tjjdd}|std|S)z Identify QuerySet of super users for the current Django instance. Raises CommandError if no super user can be found. T) is_superuserzFailure to identify superuser)robjectsfilterexistsr)Z superusersrrr_identify_super_users2sr"cCs0tjj|d}|s(td|d|dS)z Gets campaign instance for given campaign name. Paramters: - campaign_name:str specifies name of Campaign instance. ) campaignNamezCampaign {0!r} does not exist. zNo task agendas have been assigned. This message is expected if you are initializing a new campaign and have not created it in the admin panel. Create a new campaign or check for a misspelling and try again.r)r rr r!rformat) campaign_name _campaignrrr_get_campaign_instance?sr'cCs:tjj||dd|d}|dj||d|dS)zs Creates CampaignTeam instance, if it does not exist yet. Returns reference to CampaignTeam instance. d2)teamNameownerrequiredAnnotations requiredHours createdByr)r r get_or_createmembersaddsave)namer+rrZ_cteamrrr_get_or_create_campaign_teamSs r4cCstjj||||d\}}|S)zg Creates Market instance, if it does not exist yet. Returns reference to Market instance. )sourceLanguageCodetargetLanguageCode domainNamer.)rrr/) source_code target_code domain_namer+_market_unused_created_signalrrr_get_or_create_marketfs r=cCstjj|||||d\}}|S)zc Creates Meta instance, if it does not exist yet. Returns reference to Meta instance. )market corpusName versionInfosourcer.)rrr/)r> corpus_name version_inforAr+_metar<rrr_get_or_create_metavs rEc Csd}t||dttt|dddd}tt}|dD]L}d| dd | dd }|| |d}|| |qF|D]"}t||}||d ||<q|S) a Gets tasks by market, converting QuerySet to dictionary of lists of tasks. Assignment scheme: - T1 U1 U2 - T2 U3 U4 - T3 U1 U2 - T4 U3 U4 To assign this, we duplicate tasks, per market, based on REDUNDANCY Parameters: - tasks:QuerySet contains task instances to extract market keys for; - context:dict specifices CAMPAIGN_NO and REDUNDANCY. Raises: - CommandError in case of missing required key. Returns: - tasks_by_market:dict[list[str]] organises tasks by market. ) CAMPAIGN_NO REDUNDANCYz{{0}}{{1:0{0}x}}rFNidz{0}{1}-rG)_validate_required_keysr$maxlenhexgetrlistorder_bymarketSourceLanguageCodereplacemarketTargetLanguageCodelowerrkeys) rcontext required_keys format_strtasks_by_markettaskZ market_codekeyZ _single_copyrrr_get_tasks_by_markets"   r^cCszd}t|||d||f}|dkr@d||f}t|tdd|D|ddkrvd ||f}t||S) a Gets tasks-to-annotators map for given language pair. Parameters: - source_code:str specifies source language ISO 639-2/3 code; - target_code:str specifies target language ISO 639-2/3 code; - context:dict specifices REDUNDANCY and TASKS_TO_ANNOTATORS. Raises: - CommandError in case of missing required key. Returns: - tasks_map:list[int] encodes number of annotators and assigned tasks. )rGTASKS_TO_ANNOTATORSr_Nz&No TASKS_TO_ANNOTATORS mapping for {0}cSsg|] }t|qSrrN.0xrrr sz4_get_tasks_map_for_language_pair..rGrz'Bad TASKS_TO_ANNOTATORS mapping for {0})rLrPr$ LookupErrorsum ValueError)r8r9rXrY _tasks_map_msgrrr _get_tasks_map_for_language_pairs  rjc Csdt|ddN}z t|}Wn*tk rD}z t|W5d}~XYnX|W5QRSW5QRXdS)a/Loads campaign manifest data from JSON file. Parameters: - json_path:str specifies path to JSON file containing manifest. Raises: - CommandError in case of errors parsing/loading from JSON file. Returns: - campaign_data:dict[str]->any campaign data parsed from JSON file. zutf-8)encodingN)openrrr) json_pathZ json_fileZ campaign_dataexcrrr_load_campaign_manifests  rocCs|}d}d}tD]:}|dd}||rt|t|kr|}t|}q||d}d}tD]2}|dd}||rdt|t|krd|}qd||fS)zM Given some market key, identifies source and target language codes. rrKrJN)rVr rT startswithrN)r]Z target_posr8codeZcmp_coder9rrr_identify_codes_for_keys"    rrc s2d}t||t||}tt}|D]fdd|D}tjj|d}t\}} zt|| |} Wn:t t fk r} zt t | WYq$W5d} ~ XYnX|} t | } tdd| D}| |krd| |}t|t|d | D]:\}}t d || |||D]}|| ||fq qq$|S) a Map tasks to users, by market, and considering TASKS_TO_ANNOTATORS. Parameters: - tasks:QuerySet contains task instances to map to users; - usernames:list specifies user names for campaign; - context:dict specifices CAMPAIGN_NO, REDUNDANCY and TASKS_TO_ANNOTATORS. Raises: - CommandError in case of missing required key or tasks/users issues. Returns: - tasks_to_users:dict[list[tuple(User,Task)]] maps tasks to users. )rFrGr_c3s|]}|r|VqdS)N)rprar]rr #s z0_map_tasks_to_users_by_market..)Z username__inNcSsg|] }t|qSrr`rarrrrd5sz1_map_tasks_to_users_by_market..z1Mismatch of available/required tasks ({0} != {1})rIzMapping task(s) to user:)rLr^rrQrrr rrrjrergprintstrrNrfr$rziprRr)r usernamesrXrYr[tasks_to_users_mapZ _usernamesusersr8r9rh_excZ_tasks_for_current_keyZ_available_tasksZ_required_tasksriuserZtasks_for_userrrrsr_map_tasks_to_users_by_market sJ     r}TcCsbd}t||t|d}td|dtd|dt|d}|jj|d}tdt||r|jdd }td t|t |||}|D]}td |||D]\} } t| d | j t jj| |d } | st jj | |d } n| d} tjj|j| j d} | | dr(q| | dk} | rL| | dq| | dqqdS)a Processes TaskAgenda instances for campaign specified by CAMPAIGN_NAME. Parameters: - usernames:list specifies user names for campaign; - context:dict specifices CAMPAIGN_NO, REDUNDANCY, TASKS_TO_ANNOTATORS and TASK_TYPE; - only_activated:bool only include activated tasks for agenda creation. Raises: - CommandError in case of missing required key. )rFrGr_ TASK_TYPE CAMPAIGN_NAMEzIdentified Campaign {0!r}z Task type: {}r~)campaignzIdentified {} task(s)T) activatedzIdentified {} activated task(s)z[{0}]z-->)r|rr)typeName primaryIDN)rLr'rPrur$r rr rNr}rIrr!createrr/__name__ contains_tasknext_item_for_user complete_task activate_task)rxrXZonly_activatedrYr&Z _task_typerryr]r\r|ZagendaZ serialized_tZ_task_done_for_userrrr_process_campaign_agendasJs<     rc Cs`d}t|||D]F\}}zt|||}Wn:ttfk rf}ztt|WYqW5d}~XYnXtdd|D|d}t|} t |d||| } d t tt |dd dd t tt | d dd } t | D]r} | |d d |d d |d| d } tjj| d }|| jkrtd | j|j| j|qqdS)a Adds User objects to CampaignTeam instances for given language pairs. Parameters: - language_pairs:list[tuple(str, str), ...] list of language pairs; - owner:User sets the creator/owner of related campaign objects; - context:dict specifies additional context: CAMPAIGN_NAME, CAMPAIGN_NO, REDUNDANCY and TASKS_TO_ANNOTATORS. Raises: - CommandError in case of missing required key. )rrFrGr_NcSsg|] }t|qSrr`rarrrrdsz+_process_campaign_teams..rGr {{0}}{{1}}{{2:0{0}x}}{{3:0{1}x}}rFrHrJrKrusernamez {0} --> {1})rLrjrergrurvrfrPrNr4r$rMrOrrVrTrrr0allr*rr1)language_pairsr+rXrY_src_tgtrhr{Z_tasks _annotatorsZcampaign_team_objectrZuser_idrZ user_objectrrr_process_campaign_teamssH    rc Ksptf|}g}|D]X\}}t|||dd|d}t||dd|dd|dd|d }|||fq|S) a( Create Market and Metadata instances for given language pairs. Parameters: - language_pairs:list[tuple(str, str), ...] list of language pairs; - owner:User sets the creator/owner of Market and Metadata objects; - **kwargs allows to override defaults for other settings. r:Z UNDEFINED)r8r9r:r+rBrCz1.0rAZofficial)r>rBrCrAr+)dictr=rPrEr) rr+kwargs_contextZmarkets_and_metadatarrr;rDrrr_process_market_and_metadatas$      rc Cs*d}t||t}|D] \}}t|||}t|}dttt|ddddttt|ddd}t|D]} || dd| dd|d| d} t } | | d | |d  d | dd } tjj| d stjj| | d } | | || <q|q|S)a Create User instances for given language pairs. Parameters: - language_pairs:list[tuple(str, str), ...] list of language pairs; - context:dict specifies additional context: CAMPAIGN_KEY, CAMPAIGN_NO, REDUNDANCY and TASKS_TO_ANNOTATORS. ) CAMPAIGN_KEYrFrrFrHNrJrKrutf8rr)rpassword)rLrrjrNr$rMrOrPrrVrTrupdateencode hexdigestrrr r! create_userr2)rrXrYZ _credentialsrrrhrrZrrhashersecretnew_userrrr_process_userss4     rcCs$|D]}t|std|qdS)z Checks that all language codes are valid. Parameters: - codes:tuple specifies language codes. Raises: - CommandError in case of invalid language code. z%{0!r} contains invalid language code!N)r rr$)codesrqrrr_validate_language_codess rcCs(|D]}||krtd|qdS)a Checks that all required keys exist in given context dict. Parameters: - context:dict specifies context dictionary; - required_keys:tuple[str] specifies set of required keys. Raises: - CommandError in case of missing required key. z+context does not contain required key {0!r}N)rWrgr$)rXrY required_keyrrrrL%s  rLN)T)*__doc__ collectionsrrhashlibrjsonrrdjango.contrib.auth.modelsrdjango.core.management.baserCampaign.modelsr r Dashboard.modelsr r EvalData.modelsr rrrrrr"r'r4r=rEr^rjrorrr}rrrrrrLrrrrsB                 /"A C<.