U
É]ÊdáB ã @ s´ d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlmZ ddl m
Z
ddl m Z dd l
mZ dd
lmZ dd lmZ dd lmZ dd
lmZ ddlmZ ddlmZ ddlmZ ddlmZ ddlmZ ddlmZ ddlm Z ddlm Z ddlm Z ddlm Z ddlm Z ddlm!Z! ddlm"Z" ddl#m$Z$ dd l%Z%ee&d Z'ed,d d „ƒZ(ed-d d!„ƒZ)ed.d"d#„ƒZ*ed/d$d%„ƒZ+ed0d&d'„ƒZ,ed1d(d)„ƒZ-ed2d*d+„ƒZ.d S )3z>
Appraise evaluation framework
See LICENSE for usage details
é )Údatetime)Úlogin_required)Ú JsonResponse)Úredirect)Úrender)Úutc)Ú BASE_CONTEXT)Ú _get_logger)ÚCampaign)ÚSIGN_LANGUAGE_CODES)ÚDataAssessmentResult)ÚDataAssessmentTask)Ú DirectAssessmentContextResult)ÚDirectAssessmentContextTask)Ú DirectAssessmentDocumentResult)Ú DirectAssessmentDocumentTask)ÚDirectAssessmentResult)ÚDirectAssessmentTask)ÚMultiModalAssessmentResult)ÚMultiModalAssessmentTask)Ú PairwiseAssessmentDocumentResult)Ú PairwiseAssessmentDocumentTask)ÚPairwiseAssessmentResult)ÚPairwiseAssessmentTask)Ú
TaskAgenda)Ú _get_ui_langN)Únamec * C s, t ¡ }d}|rFtjj|d}| ¡ s>d}t ||¡ tdƒS |d }t d| j j
pVd¡ d}t jj| j d}|r~|j|d }|D ]†}t d
|¡ g } | ¡ D ]@}
|
¡ } | dkr´qž| | j ¡dk rÔ| }|sÞ|j}qž| |
¡ qžd } | D ]}
| |
¡pø| } qè| r‚| ¡ q‚|s>| ¡ dkr>t d ¡ t d
||¡ tdƒS |sRtj| j d}|sà|dksl|dkrŒt d¡ t d
||¡ tdƒS t d||¡ t ||| j ¡}|dkrÆt d¡ tdƒS |j | j ¡ | ¡ |}|r|sô|j}n$|j|jjkrd}t |¡ |j}t ¡ }| jdkrN| j dd¡}| j dd¡}| j dd¡}| j dd¡}| j dd¡}t d||¡ |rN|rN|rN|rNt |ƒt |ƒ }t t |ƒ¡ t t |ƒ¡ t d|||¡ | | j ¡}|j t |ƒks |j!t |ƒkrd}t |||j ¡ n8t "¡ j#t$d}t%jj&|t |ƒt |ƒ||| j d d |d t ¡ }|j| j d d \}}|s‚t d ¡ tdƒS t |d ƒ}d!}t |||¡ | '¡ }| (¡ } t ¡ } d"} d#} d$} d%d&„ d'D ƒ}!d(d&„ d)D ƒ}"d*d&„ d+D ƒ}#|j|!krd,} d#} d-} n2|j|"kr"d.} d/} d0} n|j|#kr:d.} d/} d1} |j)pDd2}$d3|$ *¡ krZd4}%nd5}%d6|$ *¡ krpd7}%d8|$ *¡ kr’d9 +| ¡} d: +| ¡} t,| ƒ\}&}'t- .t-|&¡}(d;| |j/| |j0| |j |j!|d ||d || || || | | | | fd<| j1k|j|j!| 2| j ¡|&d=œ|(–})|) 3t4¡ t5| |%|)ƒS )>z,
Direct assessment annotation view.
N©Ú campaignNameú7No campaign named "%s" exists, redirecting to dashboardÚ dashboardr ú/Rendering direct assessment view for user "%s".Ú Anonymous©Úuser©ÚcampaignúIdentified work agenda %sFú0Work agendas completed, redirecting to dashboardú- code=%s, campaign=%sú2No current task detected, redirecting to dashboardú2Identifying next task for code "%s", campaign="%s"ú/No next task detected, redirecting to dashboardú9Incompatible campaign given, using item campaign instead!ÚPOSTÚscoreÚitem_idÚtask_idÚstart_timestampÚ
end_timestampúscore=%s, item_id=%sú start=%s, end=%s, duration=%sú1Item ID %s does not match item %s, will not save!©ÚtzinfoT© r/ Ú
start_timeÚend_timeÚitemÚtaskÚ createdByÚ activatedÚ completedÚ
dateCompleted©Úreturn_completed_itemsú2No current item detected, redirecting to dashboardé
ú'completed_items=%s, completed_blocks=%sú Source textúCandidate translationú«How accurately does the above candidate text convey the original semantics of the source text? Slider ranges from Not at all (left) to Perfectly (right).c s s | ]}d |¡V qdS ©zHumanEvalFY19{0}N©Úformat©Ú.0Úx© rP ú=/var/www/rival/public_html/translation-eval/EvalView/views.pyÚ Ø s z$direct_assessment..©Z7Bc s s | ]}d |¡V qdS rJ rK rM rP rP rQ rR Ú s ©Z51Z57Z63c s s | ]}d |¡V qdS rJ rK rM rP rP rQ rR Ü s ©Z52Z58Z64úReference textú®How accurately does the above candidate text convey the original semantics of the reference text? Slider ranges from Not at all (left) to Perfectly (right).ú Candidate Aú Candidate Bú¤How accurately does candidate text B convey the original semantics of candidate text A? Slider ranges from Not at all (left) to Perfectly (right).ú°Which of the two candidate texts is more fluent? Slider marks preference for Candidate A (left), no difference (middle) or preference for Candidate B (right).Ú Úsqmz#EvalView/direct-assessment-sqm.htmlú'EvalView/direct-assessment-context.htmlZ
namedentitz.EvalView/direct-assessment-named-entities.htmlÚ referencezReference text in {}zCandidate translation in {}údirect-assessmentÚdebug)Ú active_pageÚreference_labelÚreference_textÚcandidate_labelÚcandidate_textÚpriming_question_textr0 r1 Úcompleted_blocksÚitems_left_in_blockÚsource_languageÚtarget_languageÚ debug_timesÚtemplate_debugr& Ú datask_idÚ trusted_userÚui_lang)6r Únowr
ÚobjectsÚfilterÚexistsÚLOGGERÚinfor r$ Úusernamer Úserialized_open_tasksÚget_object_instanceÚnext_item_for_userr& ÚappendÚcomplete_open_taskÚsaveÚcountr Úget_task_for_userÚ get_next_free_task_for_languageÚ
assignedToÚaddr Úmethodr. ÚgetÚfloatra ÚitemIDÚintÚidÚutcnowÚreplacer r ÚcreateÚmarketSourceLanguageÚmarketTargetLanguageÚcampaignOptionsÚlowerrL r Útranslated_textsÚ_get_lang_textsÚ
sourceTextÚ
targetTextÚGETÚis_trusted_userÚupdater r )*ÚrequestÚcodeÚ
campaign_nameÚt1r& Ú_msgÚ current_taskÚagendasÚagendaÚtasks_to_completeÚserialized_open_taskÚ open_taskÚmodifiedr= Ú next_taskÚt2r/ r0 r1 r2 r3 ÚdurationÚ current_itemÚutc_nowÚt3Úcompleted_itemsrh rj rk Út4rc re rg Ú_reference_campaignsÚ_adequacy_campaignsÚ_fluency_campaignsÚ
campaign_optsÚ html_filerp Ú_Ú
lang_textsÚcontextrP rP rQ Údirect_assessment+ sX
þ
ý ÿ
ü ÷ ÿ
ÿ ÿ ÿ ÿ
îí
r³ c & C sô t ¡ }d}|rFtjj|d}| ¡ s>d}t ||¡ tdƒS |d }t d| j j
pVd¡ d}t jj| j d}|r~|j|d }|D ]†}t d
|¡ g } | ¡ D ]@}
|
¡ } | dkr´qž| | j ¡dk rÔ| }|sÞ|j}qž| |
¡ qžd } | D ]}
| |
¡pø| } qè| r‚| ¡ q‚|s>| ¡ dkr>t d ¡ t d
||¡ tdƒS |sRtj| j d}|sà|dksl|dkrŒt d¡ t d
||¡ tdƒS t d||¡ t ||| j ¡}|dkrÆt d¡ tdƒS |j | j ¡ | ¡ |}|r|sô|j}n$|j|jjkrd}t |¡ |j}t ¡ }| jdkrh| j dd¡}| j dd¡}| j dd¡}| j dd¡}| j dd¡}| j dd¡}t d||¡ |rh|rh|rh|rht |ƒt |ƒ }t t |ƒ¡ t t |ƒ¡ t d|||¡ | | j ¡}|j t |ƒks|j!t |ƒks|j"|kr0d}t |||j ¡ n8t #¡ j$t%d }t&jj'|t |ƒt |ƒ||| j d d |d t ¡ }|j| j d d \}}|sœt d ¡ tdƒS t |d! ƒ}d"}t |||¡ | (¡ } | )¡ } t ¡ } d#} d$} d%}!|j*rêd&}!d'd(„ d)D ƒ}"d*d(„ d+D ƒ}#d,d(„ d-D ƒ}$|j|"kr.d.} d$} d/}!n2|j|#krHd0} d1} d2}!n|j|$kr`d0} d1} d3}!d4| |j+|j,|j-| |j.|j/|j0|!|j |j!|j"|j*|d!||d! | | || || | | | | fd5| j1k|j|j!| 2| j ¡d6œ}%|% 3t4¡ t5| d7|%ƒS )8z4
Direct assessment context annotation view.
Nr r r r z7Rendering direct assessment context view for user "%s".r" r# r% r' Fr( r) r* r+ r, r- r. r/ r0 r1 Ú document_idr2 r3 r4 r5 r6 r7 Tr9 rB rD rE rF rG rH rI z³How accurately does the above candidate document convey the original semantics of the source document? Slider ranges from Not at all (left) to Perfectly (right).c s s | ]}d |¡V qdS rJ rK rM rP rP rQ rR Ý s z,direct_assessment_context..rS c s s | ]}d |¡V qdS rJ rK rM rP rP rQ rR ß s rT c s s | ]}d |¡V qdS rJ rK rM rP rP rQ rR á s rU rV rW rX rY rZ r[ r` ra )rb rc rd Zreference_context_leftZreference_context_rightre rf Zcandidate_context_leftZcandidate_context_rightrg r0 r1 r´ ÚisCompleteDocumentrh ri rj rk rl rm r& rn ro r^ )6r rq r
rr rs rt ru rv r r$ rw r rx ry rz r& r{ r| r} r~ r r r€ r r‚ r rƒ r. r„ r… ra r† r‡ rˆ Ú
documentIDr‰ rŠ r r r‹ rŒ r rµ r’ ÚsourceContextLeftÚsourceContextRightr“ ÚtargetContextLeftÚtargetContextRightr” r• r– r r )&r— r˜ r™ rš r& r› rœ r rž rŸ r r¡ r¢ r= r£ r¤ r/ r0 r1 r´ r2 r3 r¥ r¦ r§ r¨ r© rh rj rk rª rc re rg r« r¬ r r² rP rP rQ Údirect_assessment_context$ sZ
þ
ý ÿ
ü ÿ þý÷ ÿ
ÿÿ ÿ ÿ ÿ
é
r» c > C sÐ t ¡ }d}|rFtjj|d}| ¡ s>d}t ||¡ tdƒS |d }t d| j j
pVd¡ d}t jj| j d}|r~|j|d }|D ]†}t d
|¡ g } | ¡ D ]@}
|
¡ } | dkr´qž| | j ¡dk rÔ| }|sÞ|j}qž| |
¡ qžd } | D ]}
| |
¡pø| } qè| r‚| ¡ q‚|s>| ¡ dkr>t d ¡ t d
||¡ tdƒS |sRtj| j d}|sà|dksl|dkrŒt d¡ t d
||¡ tdƒS t d||¡ t ||| j ¡}|dkrÆt d¡ tdƒS |j | j ¡ | ¡ |}|r|sô|j}n$|j|jjkrd}t |¡ |j}t ¡ }d }d }d}| jdkr4| j dd¡}| j dd¡}| j dd¡}| j dd¡}| j dd¡}| j dd¡}t | j dd¡dkƒ}t d ||¡ t d |||¡ƒ |r4|r4|r4|r4t |ƒt |ƒ }t !t |ƒ¡ t !t |ƒ¡ t d |||¡ |j"| j d d \}}} |j#|kr |j$t%|ƒkr¨|j&t%|ƒkr¨t '¡ j(t)d } t*jj+|t |ƒt |ƒ||| j d d!| d" t d# ||¡ƒ d!}q4d} | D ]>} | s¾q°| j,j$t%|ƒkr°| j,j&t%|ƒkr°| } qðq°| r\| j-} || _-t |ƒ| _.t |ƒ| _/t '¡ j(t)d } | | _0| ¡ d$ ||| |¡}t !|¡ t |ƒ d!}nÀd }!|D ]0}"|"j$t%|ƒkrd|"j&t%|ƒkrd|"}! q–qd|!røt '¡ j(t)d } t*jj+|t |ƒt |ƒ|!|| j d d!| d" d% ||¡}t !|¡ t |ƒ d!}n$d&}d' ||j$¡}t !|¡ t |ƒ nt d( |j#|¡ƒ d)}t ¡ }#|j1pFd 2¡ }$| "| j ¡\}}%}&}'}} }(|s~t d*¡ tdƒS g })d}*t3|| ƒD ]¨\}"} t | o¨| j-d+kƒt |"j&|j&kƒ| rÄ| j-nd+d,œ}+d-|$kr(|"j4r(|"j$|*j$d. kr(|" j$d.7 _$|" ¡ d/ |"| j j
¡}t |ƒ t |¡ |) |+¡ |"}*qd0}t ||%|&¡ | 5¡ },| 6¡ }-t ¡ }.d1}/d1}0d2}1d3}2d4|$k}3d5|$k}4d6|$k}5d7|$k}6d8|$k}7d9}8d:|$krºd!}4d!}6d!}7d;}8|4rê|,t7krÒd<}/d=}1|-t7kræd<}0d>}2nd }4d? t8|ƒd. |,|-|/¡d@dAg}9dBd@dC |-|,¡g}:|7r8|9dd.… }9|:dd.… }:|3rbd},dD t8|ƒd. |-¡g}9dEg}:d}2d:|$kr¦dF|$krŒdG t8|ƒd. ¡g}9ndH t8|ƒd. ¡g}9dIg}:dJ|$krîd!}5|9dKdLg7 }9| 9¡ dMkrÞ|9d+ dN7 < |9d+ dO7 < d-|$kr|9dKdPg7 }9dQ|j$|j&|j#|&|(t8|ƒ|' |,|-|/|0|| |#| |.|# |.| fdR| j:k|j|j&| ;| j ¡|7|5|4|3|6|8dSœ};|rš||dTœ}<|; <|<¡ |; |;ƒS t3||)ƒ|1|2|9|:dUœ}=|; <|=¡ |; {}ú- entire candidate document translation in {0} (right column) convey the original semantics of the source document in {1} (left column)? zËBelow you see a document with {0} sentences in {1}. Score each sentence in the document context. You may revisit already scored sentences and update their scores at any time by clicking at a source text.zPlease score the overall document quality (you can score the whole document only after scoring all individual sentences first).Z text2signu‡ Unten sehen Sie ein Dokument mit {0} Sätzen auf Deutsch (linke Spalten) und die entsprechenden möglichen Ãœbersetzungen in Deutschschweizer Gebärdensprache (DSGS) (rechte Spalten). Bewerten Sie jede mögliche Ãœbersetzung des Satzes im Kontext des Dokuments. Sie können bereits bewertete Sätze jederzeit durch Anklicken eines Quelltextes erneut aufrufen und die Bewertung aktualisieren.u‰ Unten sehen Sie ein Dokument mit {0} Sätzen in Deutschschweizer Gebärdensprache (DSGS) (linke Spalten) und die entsprechenden möglichen Ãœbersetzungen auf Deutsch (rechte Spalten). Bewerten Sie jede mögliche Ãœbersetzung des Satzes im Kontext des Dokuments. Sie können bereits bewertete Sätze jederzeit durch Anklicken eines Eingabevideos erneut aufrufen und die Bewertung aktualisieren.u¥ Bitte bewerten Sie die Ãœbersetzungsqualität des gesamten Dokuments. (Sie können das Dokument erst bewerten, nachdem Sie zuvor alle Sätze einzeln bewertet haben.)Ziwslt2022dialectsrcz\Please take into consideration the following aspects when assessing the translation quality:a4
- The document is part of a conversation thread between two speakers, and each segment starts with either "A:" or "B:" to indicate the speaker identity.
- Some candidate translations may contain "%pw" or "% pw", but since they correspond to partial words in the speech they should not be considered as errors during evaluation.
- Please ignore the lack of capitalization and punctuation. Also, please ignore "incorrect" grammar and focus more on the meaning: these segments are informal conversations, so grammatical rules are not so strict.
Zaebzi- The original source is Tunisian Arabic speech. There may be some variation in the transcription.
z
aj - The source texts come from transcribed video content published on YouTube.
- Transcribed sentences have been split into segments based on pauses in the audio. It may happen that a single source sentence is split into multiple segments.
- Please score each segment (including very short segments) individually with regard to the source segment and the surrounding context.
- Take into account both grammar and meaning when scoring the segments.
- Please pay attention to issues like repeated or new content in the candidate translation, which is not present in the source text.
zdirect-assessment-documentra )rb r0 r1 r´ rh Ú total_blocksri rj rk Úsource_item_typeÚtarget_item_typerl rm r& rn ro r] ZspeechrË rÊ Ústatic_contextrp ©ZsavedÚ error_msg)Úitemsrc re Úpriming_question_textsÚdocument_question_textsz(EvalView/direct-assessment-document.html)@r rq r
rr rs rt ru rv r r$ rw r rx ry rz r& r{ r| r} r~ r r r€ r r‚ r rƒ r. r„ ÚboolÚprintrL r… ra Únext_document_for_userr¶ r† r‡ rˆ r‰ rŠ r r r‹ r< r/ r: r; rA rŽ r Úziprµ rŒ r r ÚlenÚmarketSourceLanguageCoder” r• r– r r r )>r— r˜ r™ rš r& r› rœ r rž rŸ r r¡ r¢ r= r£ r¤ r½ Ú
item_savedrÔ r/ r0 r1 r´ r2 r3 r¥ r¦ Ú block_itemsÚ
block_resultsr§ Úcurrent_resultÚresultZ
prev_scoreÚ
found_itemr< r¨ r® r© rh Úcompleted_items_in_blockrÏ Ú block_scoresZ
_prev_itemÚ item_scoresrj rk rª rÐ rÑ rc re Úmonolingual_taskZsign_translationZspeech_translationrÒ Úuse_sqmZ ui_languagerÖ r× r² Ú ajax_contextÚ page_contextrP rP rQ Údirect_assessment_document s´
þ
ý ÿ
ÿü
ÿü ÿ÷ ÿ
ÿ
÷ ÿ
ÿ ÿ
ÿÿÿ
ø
ý
ÿþý ÿ
ø
ô þü
üÿ ÿ
úÿ
úÿ ÿ
þ
ÿ
þ
é
û
rë c C s6 t ¡ }d}|rFtjj|d}| ¡ s>d}t ||¡ tdƒS |d }t d| j j
pVd¡ d}t jj| j d}|r~|j|d }|D ]†}d
} t d |¡ g }
| ¡ D ]@} |
¡ } | dkr¸q¢| | j ¡dk rØ| }|sâ|j}q¢|
| ¡ q¢|
D ]}
| |
¡pø| } qè| r‚| ¡ q‚|s>| ¡ dkr>t d ¡ t d
||¡ tdƒS |sRtj| j d}|sä|dksl|dkrŒt d¡ t d
||¡ tdƒS d}t |||¡ t ||| j ¡}|dkrÊt d¡ tdƒS |j | j ¡ | ¡ |}|r |sø|j}n$|j|jjkr d}t |¡ |j}t ¡ }| jdkrR| j dd¡}| j dd¡}| j dd¡}| j dd¡}| j dd¡}t d||¡ |rR|rR|rR|rRt |ƒt |ƒ }t t |ƒ¡ t t |ƒ¡ t d|||¡ | | j ¡}|j t |ƒks|j!t |ƒkrd}t |||j ¡ n8t "¡ j#t$d}t%jj&|t |ƒt |ƒ||| j d
d |d t ¡ }|j| j d d \}}|s†t d ¡ tdƒS t |d ƒ}d!}t |||¡ | '¡ }| (¡ } t ¡ } d"|j)|j*|j+|j |j!|d ||d || || || | | | | fd#| j,k|j|j!| -| j ¡d$œ} | .t/¡ t0| d%| ƒS )&z1
Multi modal assessment annotation view.
Nr r r r z3Rendering multimodal assessment view for user "%s".r" r# r% Fr' r( r) r* r+ r, r- r. r/ r0 r1 r2 r3 r4 r5 z2Item ID %s does not match item %s, will not save!r7 Tr9 rB rD rE rF zmultimodal-assessmentra )rb rd rf Z image_urlr0 r1 rh ri rj rk rl rm r& rn ro z#EvalView/multimodal-assessment.html)1r rq r
rr rs rt ru rv r r$ rw r rx ry rz r& r{ r| r} r~ r r r€ r r‚ r rƒ r. r„ r… ra r† r‡ rˆ r‰ rŠ r r r‹ rŒ r r’ r“ ÚimageURLr” r• r– r r ) r— r˜ r™ rš r& r› rœ r rž r¢ rŸ r r¡ r= r£ r¤ r/ r0 r1 r2 r3 r¥ r¦ r§ r¨ r© rh rj rk rª r² rP rP rQ Úmultimodal_assessment s
þ
ÿ
ü ÷ ÿ
ñ
rí c ( C s´ t ¡ }d}|rFtjj|d}| ¡ s>d}t ||¡ tdƒS |d }t d| j j
pVd¡ d}t jj| j d}|r~|j|d }|D ]†}t d
|¡ g } | ¡ D ]@}
|
¡ } | dkr´qž| | j ¡dk rÔ| }|sÞ|j}qž| |
¡ qžd } | D ]}
| |
¡pø| } qè| r‚| ¡ q‚|s>| ¡ dkr>t d ¡ t d
||¡ tdƒS |sRtj| j d}|sà|dksl|dkrŒt d¡ t d
||¡ tdƒS t d||¡ t ||| j ¡}|dkrÆt d¡ tdƒS |j | j ¡ | ¡ |}|r|sô|j}n$|j|jjkrd}t |¡ |j}t ¡ }| jdkrr| j dd¡}| j dd¡}| j dd¡}| j dd¡}| j dd¡}| j dd¡}t d |||¡ƒ t d|||¡ |rr|rr|rr|rrt |ƒt |ƒ }t t |ƒ¡ t t |ƒ¡ t d|||¡ | | j ¡}|j!t"|ƒks"|j#t"|ƒkr8d }t |||j!¡ n:t $¡ j%t&d }t'jj(||t |ƒt |ƒ||| j d d |d
t ¡ }|j| j d d \}}|s¦t d!¡ tdƒS t"|d" ƒ}d#}t |||¡ | )¡ } | *¡ } t ¡ } d$} d%} d&}!d'}"| +¡ rúd(}"| ,¡ \}#}$d)| |j-| .¡ | /¡ | |#|!|$|"|j!|j#|d"||d" | | || || | | | | fd*| j0k|j|j#| 1| j ¡d+œ}%|% 2t3¡ |j4pŽd,}&d-|& 5¡ kr¤d.}'nd/}'t6| |'|%ƒS )0z5
Pairwise direct assessment annotation view.
Nr r r r z8Rendering pairwise direct assessment view for user "%s".r" r# r% r' Fr( r) r* r+ r, r- r. r/ Úscore2r0 r1 r2 r3 z#score1={0}, score2={1}, item_id={2}ú score1=%s, score2=%s, item_id=%sr5 r6 r7 T©
Úscore1rî r: r; r< r= r> r? r@ rA rB rD rE rF rG zCandidate translation (1)zCandidate translation (2)zoHow accurately does each of the candidate text(s) below convey the original semantics of the source text above?zvHow accurately does each of the candidate text(s) below convey the original semantics of the bolded source text above?zpairwise-assessmentra )rb rc rd Ú context_leftÚ
context_rightre rf Úcandidate2_labelÚcandidate2_textrg r0 r1 rh ri rj rk rl rm r& rn ro r\ r] z%EvalView/pairwise-assessment-sqm.htmlz!EvalView/pairwise-assessment.html)7r rq r
rr rs rt ru rv r r$ rw r rx ry rz r& r{ r| r} r~ r r r€ r r‚ r rƒ r. r„ rÙ rL r… ra r† r‡ rˆ r‰ rŠ r r r‹ rŒ r Ú has_contextÚtarget_texts_with_diffsÚ segmentTextrò ró r” r• r– r rŽ r r )(r— r˜ r™ rš r& r› rœ r rž rŸ r r¡ r¢ r= r£ r¤ rñ rî r0 r1 r2 r3 r¥ r¦ r§ r¨ r© rh rj rk rª rc Úcandidate1_labelrô rg Úcandidate1_textrõ r² r® r¯ rP rP rQ Úpairwise_assessmentË s>
þ
ý ÿ
ü ö
ÿ
ÿ
ÿý
ë
rû c ( s t ¡ }d}|rFtjj|d}| ¡ s>d}t ||¡ tdƒS |d }t d| j j
pVd¡ d}t jj| j d}|r~|j|d }|D ]†}t d
|¡ g } | ¡ D ]@}
|
¡ } | dkr´qž| | j ¡dk rÔ| }|sÞ|j}qž| |
¡ qžd } | D ]}
| |
¡pø| } qè| r‚| ¡ q‚|s>| ¡ dkr>t d ¡ t d
||¡ tdƒS |sRtj| j d}|sà|dksl|dkrŒt d¡ t d
||¡ tdƒS t d||¡ t ||| j ¡}|dkrÆt d¡ tdƒS |j | j ¡ | ¡ |}|r|sô|j}n$|j|jjkrd}t |¡ |j}t ¡ }| jdkr~| j dd¡}| j dd¡}| j dd¡}| j dd¡}| j dd¡}| j dd¡}d |||¡}t |¡ t |ƒ |dkr´t dƒ nÊ|r~|r~|r~t |ƒt |ƒ }t t |ƒ¡ t t |ƒ¡ t d|||¡ | | j ¡}|j!t"|ƒks.|j#t"|ƒkrDd }t |||j!¡ n:t $¡ j%t&d }t'jj(||t |ƒt |ƒ||| j d d |d
t ¡ }|j| j d d \}}|s²t d!¡ tdƒS t"|d" ƒ}d#}t |||¡ | )¡ } | *¡ } t ¡ } d$} d%} d&j | | d'd(d)j | | d'g}!d*g}"d+j | | d'g}#d,d-d.d/g}$t+| ,¡ ƒ}%|j-pFd0 .¡ ‰ d1ˆ k}&t/‡ fd2d3„d4D ƒƒrˆd}$d}#|"d d5d… |"d< d6| | |%|!|"|#|$|&|j!|j#|j0|j1|j2|d"||d" | | || || | | | | fd7| j3k|j|j#| 4| j ¡d8œ}'|' 5t6¡ t7| d9|'ƒS ):z1
Direct data assessment annotation view.
Nr r r r r! r" r# r% r' Fr( r) r* r+ r, r- r. r/ Úrankr0 r1 r2 r3 zscore={} rank={} item_id={}z!No score provided, will not save!r5 r6 r7 T)
r/ rü r: r; r< r= r> r? r@ rA rB rD rE rF rG Z Translationz?You are presented a fragment of a document in {src} and {trg}. )ÚsrcZtrgzáPlease judge the quality of the translations between the documents on a scale from poor (left) to perfect (right), taking in to account aspects like adequacy, fluency, writing ability, orthography, style, misalignments, etc. a Please consider these aspects in both the {src} and {trg} part. For example, poor fluency in the {src} fragment is a problem too. While you may use the context from the other sentences in the document, the translations need to be correct at the sentence level.z¤Question #1: What is the quality of the translations, taking in to account aspects like adequacy, fluency, writing ability, orthography, style, misalignments, etc.?z„Question #2: Do you think any of the sentences ({src} or {trg}) were created by machine translation, rather than written by a human?)z Definitely machine-translatedrÈ )zPossibly machine-translatedé )zPossibly human-writtené )zDefinitely human-writtené r\ r] c 3 s | ]}|ˆ kV qd S )NrP )rN Úopt©r® rP rQ rR { s z"data_assessment..)ZdisablemtlabelZ
disablemtranké
zdata-assessmentra )rb Ú source_labelÚ target_labelÚ
parallel_dataÚtop_question_textÚscore_question_textÚrank_question_textÚranksr] r0 r1 Zdocument_domainZ
source_urlZ
target_urlrh ri rj rk rl Z
show_debugr& rn ro z EvalView/data-assessment.html)8r rq r
rr rs rt ru rv r r$ rw r rx ry rz r& r{ r| r} r~ r
r r€ r r‚ r rƒ r. r„ rL rÙ r… ra r† r‡ rˆ r‰ rŠ r r r‹ rŒ r ÚlistÚget_sentence_pairsrŽ r ÚanyÚdocumentDomainÚ sourceURLÚ targetURLr” r• r– r r )(r— r˜ r™ rš r& r› rœ r rž rŸ r r¡ r¢ r= r£ r¤ r/ rü r0 r1 r2 r3 r¥ r¦ r§ r¨ r© rh rj rk rª r r r r r r
r rè r² rP r rQ Údata_assessment® sf
þ
ý ÿ
ü ö
ÿ
ÿ üúÿ ýÿ ü
é
r c = C s¢ t ¡ }d}|rFtjj|d}| ¡ s>d}t ||¡ tdƒS |d }t d| j j
pVd¡ d}t jj| j d}|r~|j|d }|D ]†}t d
|¡ g } | ¡ D ]@}
|
¡ } | dkr´qž| | j ¡dk rÔ| }|sÞ|j}qž| |
¡ qžd } | D ]}
| |
¡pø| } qè| r‚| ¡ q‚|s>| ¡ dkr>t d ¡ t d
||¡ tdƒS |sRtj| j d}|sà|dksl|dkrŒt d¡ t d
||¡ tdƒS t d||¡ t ||| j ¡}|dkrÆt d¡ tdƒS |j | j ¡ | ¡ |}|r|sô|j}n$|j|jjkrd}t |¡ |j}t ¡ }d }d }d}| jdkrZ| j dd¡}| j dd¡}| j dd¡}| j dd¡}| j dd¡}| j dd¡}| j dd¡}t | j dd¡d kƒ}t d |||¡ t d ||||¡ƒ |rZ|rZ|rZ|rZt |ƒt |ƒ }t !t |ƒ¡ t !t |ƒ¡ t d |||¡ |j"| j d d \}} } |j#|krD|j$t%|ƒkr¼|j&t%|ƒkr¼t '¡ j(t)d!} t*jj+||t |ƒt |ƒ||| j d d"| d#
t d$ ||¡ƒ d"}qZd} | D ]>} | sÒqÄ| j,j$t%|ƒkrÄ| j,j&t%|ƒkrÄ| } qqÄ| r€| j-}!| j.}"|| _-|| _.t |ƒ| _/t |ƒ| _0t '¡ j(t)d!} | | _1| ¡ d% |||!||"|¡}t !|¡ t |ƒ d"}nÂd }#| D ]0}$|$j$t%|ƒkrˆ|$j&t%|ƒkrˆ|$}# qºqˆ|#r t '¡ j(t)d!} t*jj+||t |ƒt |ƒ|#|| j d d"| d#
d& ||¡}t !|¡ t |ƒ d"}n$d'}d( ||j$¡}t !|¡ t |ƒ nt d) |j#|¡ƒ d*}t ¡ }%| "| j ¡\}}&}'}(} } })|s”t d+¡ tdƒS g }*t2| | ƒD ]f\}$} |$ 3¡ \}+},t | oÆ| j-d,kƒt |$j&|j&kƒ| râ| j-nd,| rð| j.nd,|+|,d-œ}-|* |-¡ q¢d.}t ||&|'¡ | 4¡ }.| 5¡ }/t ¡ }0d/}1d0}2d1}3d2 t6| ƒ|.|/¡d3d4g}4d5d3d6 |/|.¡g}5|j7ptd 8¡ }6d7|6k}7d8|6k}8d9|6k}9|8r°|4dd:… }4|5dd:… }5|7rÞd}.d; t6| ƒd: |/¡g}4d}3d?|j$|j&|j#|'|)t6| ƒ|( |.|/|| |%| |0|% |0| fd@| j9k|j|j&| :| j ¡|7|8|9dAœ}:|rj||dBœ};|: ;|;¡ |: ;t<¡ t=|:ƒS t2| |*ƒ|1|2|3|4|5dCœ}<|: ;|<¡ |: ;t<¡ t>| dD|:ƒS )Ez>
Pairwise direct assessment document annotation view.
Nr r r r r¼ r" r# r% r' Fr( r) r* r+ r, r- r\ r. rñ rî r0 r1 r´ r2 r3 r½ r¾ rï z9Got request score1={0}, score2={1}, item_id={2}, ajax={3}r5 r¿ r7 Trð rÁ z-Item {} (itemID={}) updated {}->{} and {}->{}r rà rÄ rÅ rÆ rD rÇ )r@ r¦ rñ rî rú rõ rF rG zCandidate translation (A)zCandidate translation (B)aZ Below you see a document with {0} sentences in {1} (left columns) and their corresponding candidate translations from two different systems in {2} (right columns). Score each candidate sentence translation in the system's document context. You may revisit already scored sentences and update their scores at any time by clicking at a source text.rÍ z°How accurately does the candidate text for each system (right column, in bold) convey the original semantics of the source text (left column) in the system's document context? zœPlease score the overall document translation quality for each system (you can score the whole documents only after scoring all individual sentences first).rÎ rÊ r] rÌ rÈ zóBelow you see two documents, each with {0} sentences in {1}. Score each sentence in both documents in their respective document context. You may revisit already scored sentences and update their scores at any time by clicking at a source text.zšPlease score the overall quality of each document (you can score the whole document only after scoring all individual sentences from all documents first).z
Sentence Az
Sentence Bz pairwise-assessment-documentra )rb r0 r1 r´ rh rÏ ri rj rk rl rm r& rn ro rÊ r] rÒ rÓ )rÕ rc rù rô rÖ r× z*EvalView/pairwise-assessment-document.html)?r rq r
rr rs rt ru rv r r$ rw r rx ry rz r& r{ r| r} r~ r r r€ r r‚ r rƒ r. r„ rØ rÙ rL r… ra rÚ r¶ r† r‡ rˆ r‰ rŠ r r r‹ r< rñ rî r: r; rA rÛ r÷ rŒ r rÜ rŽ r r” r• r– r r r )=r— r˜ r™ rš r& r› rœ r rž rŸ r r¡ r¢ r= r£ r¤ r½ rÞ rÔ rñ rî r0 r1 r´ r2 r3 r¥ r¦ rß rà r§ rá râ Z prev_score1Z prev_score2rã r< r¨ r© rh rä rÏ rå Z_candidate1_textZ_candidate2_textræ rj rk rª rc rù rô rÖ r× r® rç rè rÒ r² ré rê rP rP rQ Ú pairwise_assessment_documentŸ sV
þ
ÿ
ý ÿ
ÿÿü
ÿü ÿö ÿ
ÿ
ö ÿ
ÿ ÿ
ÿÿÿ
ø
ú úö þû
üÿ ÿ
ï
ú
r )NN)NN)NN)NN)NN)NN)NN)/Ú__doc__r Ú django.contrib.auth.decoratorsr Ú django.httpr Údjango.shortcutsr r Zdjango.utils.timezoner ÚAppraise.settingsr ÚAppraise.utilsr ÚCampaign.modelsr
ÚDashboard.modelsr ÚEvalData.modelsr r
r r r r r r r r r r r r r ZDashboard.viewsr r Ú__name__ru r³ r» rë rí rû r r rP rP rP rQ Ú sf
y y 1 c q